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ABSTRACT 



This report examines performance evaluations of state 
loan/scholarship programs and direct support for medical education reform 
programs for the education and training of health care professionals. It is 
based on a survey of all administrators of state scholarship and loan 
repayment programs and a mail survey of officials in the 34 states which have 
massed medical education reform laws. Part 1 examines efforts to evaluate 
state scholarship and loan repayment programs and Part 2 reports on 
evaluations of state medical education reforms. Overall, the study found that 
most scholarship and loan repayment programs and other medical education 
reforms enacted by state legislatures have not been evaluated to determine 
their effectiveness. Concerning scholarship and loan repayment programs, the 
study found only 20 percent of programs have some kind of report that 
documents program effectiveness although another 20 percent are planning 
evaluations. Concerning medical education reforms, only 10 of the 34 states 
have produced acceptable evaluations of recent state-directed or 
multi-institutional medical education reform initiatives. Most state laws 
carry no appropriation to evaluate, nor do they contain measures to enforce a 
new program's effectiveness. It is urged that some measure of accountability 
be demanded of programs, to help state legislators as they set priorities for 
short- and long-term issues, solutions, and allocation of funds. Supporting 
data appear in tables and figures. Appendices include: the survey 
questionnaire, a directory of programs. National Health Service Corps (NHSC) 
reporting requirements, legislation, and other compilations. (BF) 
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Executive Summary 



The role of state government in supporting the education and training of health 
professionals is well established. Traditionally, states have supported loan and scholarship 
programs for medical students and physicians in training and, for almost 50 years, most 
states have provided some level of financial support for medical education. 

This two-part report examines the extent to which performance evaluations exist for these 
two state strategies to determine whether they improve the supply of primary care 
professionals, particularly in medically underserved areas, and whether these evaluations 
have been useful to state policymakers. Part I examines efforts to evaluate state scholarship 
and loan repayment programs, and Part II reports on evaluations of state medical education 
reforms. 



This study found that most scholarship and loan repayment programs and other medical 
education reforms enacted by state legislatures have not been evaluated to determine their 
effectiveness. 



Scholarship and Loan Repayment Programs. Most state legislatures do not require 
scholarship and loan repayment programs to be evaluated. About three-fifths of the 111 
programs identified have not conducted or are not planning to conduct an evaluation in the 
near future. Most officials of these programs reported that the programs are either too new 
or lack the resources — time, staff and/or money — to conduct such evaluations. Only one 
out of five programs have some kind of report that documents its effectiveness. The states 
with program evaluations include: Arizona, Arkansas, Delaware, Georgia, Idaho, Illinois, 
Michigan, Nebraska, New Jersey, New York, Oklahoma, Tennessee, Texas, West Virginia 
and Wisconsin. Another 20 percent of the programs are planning evaluations in the near 
future. 

Medical Education Reforms. Most medical education reforms enacted by state legislatures 
also have not been evaluated. While 34 states passed laws between 1985 and 1993 
requiring various institutions to implement medical education reforms, officials from just 10 
of those states produced acceptable evaluations of recent state-directed or multi-institutional 
initiatives. Most of these evaluations were conducted very recently, and thus little is known 
about whether they have had any impact on medical education funding or other program 
changes. 

Despite the significant action taken by state legislatures in the past 10 years to create and 
improve loan repayment and medical education initiatives, much less attention has been 
placed on ensuring that these initiatives are effectively monitored and evaluated. Most laws 
carry no appropriation to evaluate nor contain measures to enforce a new program's 
effectiveness, therefore providing the state minimal evidence of its success. Thus, based on 
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the study's findings, the authors recommend that state legislatures or health departments 
require evaluations to be conducted of these initiatives on a routine basis to ensure 
accountability. 
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State Scholarship and Loan 
Repayment Programs 



Rationale and Purpose 

Recruiting and retaining a sufficient number of primary care providers in medically 
underserved rural and inner cities is a significant challenge. Most recently, the U.S. Public 
Health Service estimates that more than 5,300 additional primary care practitioners are 
needed to eliminate 2,677 federally designated primary care health professional shortage 
areas (HPSAs) in order to achieve a ratio of one provider for every 3,500 citizens. 

For nearly 25 years the National Health Service Corps (NHSC), and in the past 10 years 
numerous state scholarship and loan repayment initiatives, have attempted to remedy this 
problem by placing primary care providers in needy areas. These federal and state 
programs offer scholarships and financial assistance for educational loans and debts in 
return for the recipient agreeing to practice in designated geographic areas that are 
underserved and in designated specialties of care (most, often primary care) for a specified 
time period. 

Recently, state legislatures have enacted numerous laws to create and improve scholarship 
and loan repayment programs. In 1995 alone, five states established new programs, two 
states appropriated additional funds, two states toughened obligation service standards, and 
seven states made administrative or minor adjustments to their programs. 

Although states have most often used scholarship and loan repayment programs as the 
strategy for increasing the number of primary care providers working in underserved areas, 
these programs in general have been criticized because many health care providers do not 
remain in HPSAs beyond their service obligation. Why is this so? Are these initiatives 
being evaluated to determine their effectiveness? Given tight state budgets, are state 
legislatures making these programs accountable by mandating outcome data as a condition 
of appropriations? 

This assessment of state loan repayment and scholarship programs was conducted to: 

• Document the existence of various state funded scholarship and loan repayment 
programs; 

• Determine the extent to which they are being evaluated; and 

• Identify lessons to be learned from evaluations of the various programs. 
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The study focuses on program evaluations because the effectiveness of these state 
scholarship and loan repayment programs is largely unknown. Though these programs 
have recruited health professionals to underserved areas, it has been difficult for states to 
determine if these professionals are practicing in these areas beyond their service obligation 
without the benefit of effective monitoring and evaluation. 



Methodology 

Researchers sent a brief questionnaire to all administrators of state scholarship and loan 
repayment programs for primary care students who were identified by the Association of 
American Medical Colleges in its October 1995 publication; State and Other Loan 
Repayment/Forgiveness and Scholarship Programs , Second Edition . The administrators 
were asked the following: 

• To identify all state supported scholarship and loan repayment programs in the state; 

• To provide information on the programs, as well as contact names and phone numbers; 

• To report on whether evaluations have been or are planning to be conducted on the 
programs they identified; and 

® To specify the reasons for not conducting evaluations (if applicable). 

Researchers learned from program administrators about other such programs in the state 
and sent the same questionnaire to those administrators identified by their colleagues. Thus, 
researchers relied upon information first from the Association of American Medical Colleges 
and then program administrators to identify state scholarship and loan repayment programs 
and provide certain programmatic information. (See Appendix l-A for a copy of the 
questionnaire.) 

For those programs that had been evaluated, administrators were asked to mail the 
evaluation to researchers for further study. The term, "evaluation/ 7 was defined broadly to 
mean any report or document that describes the progress or effectiveness of state-supported 
scholarship and loan programs in improving access to primary care in medically underserved 
areas. 



After reviewing the evaluations, researchers interviewed 14 administrators in 13 states — 
Arizona, Arkansas, Delaware, Idaho, Illinois, Michigan, Nebraska, New Jersey, Oklahoma, 
Tennessee, Texas, West Virginia and Wisconsin — to learn more about the evaluation 
process. For all of these states, except Oklahoma and Texas, each administrator only talked 
about one program evaluation. In Oklahoma, the administrator was interviewed about the 
evaluations of three state programs. Two administrators were interviewed in Texas because 
the programs are managed by two different offices. 

Researchers asked program administrators: 

• When the evaluations were first conducted; 

• The reasons for conducting the evaluations; 

• What information was collected; 

• The data collection systems used; 

• The cost and personnel needed to conduct the evaluations; 

• The frequency with which the evaluations are updated; 

• The usefulness of the evaluations; and 

• Their advice to other program administrators. 
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Researchers achieved a response rate of 100 percent. After much persistence, researchers 
corresponded with administrators from the statewide programs by sending them two 
mailings, making two sets of follow-up phone calls and interviewing those administrators 
that had conducted evaluations. 

However, there were significant constraints to conducting this study. First, researchers 
experienced great difficulty in identifying all of the state scholarship and loan repayment 
programs because of their large numbers and the different agencies that administer them 
throughout the states. Therefore, researchers are only certain that 1 1 1 state scholarship and 
loan programs are in operation. Second, researchers experienced further difficulties 
determining the years that the programs had been in operation given that many of the 
programs were created a year or two before they actually became operational. Again, 
researchers relied on the Association of American Medical Colleges and, in certain cases, 
on program administrators to provide this information. Third, researchers depended on 
program administrators to determine if they had conducted an evaluation. For some 
programs, there is a fine line between an evaluation and statistics on program participants. 
Thus, though this study finds that few state legislatures require programs to be formally 
evaluated, many program administrators collect data on program participants but do not 
publish this information in a formal report. 



General Findings 

1 . State scholarship and loan repayment programs are a relatively new yet widespread 
initiative. 

At least 1 1 1 state-supported scholarship and loan repayment programs exist in 47 states. 
(See Appendix l-B for a directory of the programs.) A variety of state agencies administer 
these programs which receive state funding often in combination with federal and local 
funds. Nine statewide programs (out of the 111), however, receive no state funds. Their 
funding is as follows: 

• California State Loan Repayment Program: federal and nonprofit funds; 

• Colorado Health Professions Loan Repayment Program: federal and community funds; 

• Florida Nursing Loan Forgiveness Program and Florida Scholarship Program: licensure 
fee funds; 

• Rural Kentucky Medical Scholarship Fund and Establish Practice Grant Program: 
private foundation funds; 

• Montana Rural Physician Incentive Program: trust fund from student fees; 

• Tennessee Health Access Incentive Program: unclaimed property funds; and 

• West Virginia Medical Student Loan Program: medical student fees. 

These programs are included in this study because all of them, except Kentucky's, are 
administered by state agencies and serve a statewide audience. The programs in Kentucky 
are included because they are statewide programs with many of the same features as other 
state scholarship and loan repayment programs. 

State legislatures have played an integral role in creating and appropriating state funds to 
these programs within recent years. Over the past 10 years, states have moved away from 
supporting traditional scholarship programs, which have appeared to many lawmakers less 
effective in increasing the supply of primary care physicians in underserved areas. Instead, 
most states have a larger commitment to loan repayment programs that require an 
immediate service commitment in a HPSA in exchange for a school loan. In addition to 
physicians, these newer programs are targeted toward nurses, physician assistants and other 
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providers that are needed in HPSAs. For these programs, providers typically must practice 
in HPSAs one year for every year a loan is received. 

Most of these state scholarship and loan repayment programs are young. More than half of 
the programs (63) have been in operation for 5 years or less, and over three-fourths of them 
(87) are just 1 0 years of age or younger. 



Table 1 

State Scholarship and Loan Repayment Programs by Years in Existence 


Years in Existence 




Number of Programs 


1 - 5 years 




63 


6-10 years 




24 


11-20 years 




9 


21 - 30 years 




5 


31-40 years 




0 


41 + years 




5 


Information Unavailable 




_5 


TOTAL PROGRAMS 




111 



2. Though scholarship and loan repayment programs among the states share the same 
goal, they differ in terms of funding, program size, organizational structure and 
administration. 

All state scholarship and loan repayment programs aim to increase the number of providers 
practicing in shortage areas. However, it is challenging to identify each state scholarship 
and loan repayment program because different agencies administer the programs. Higher 
education boards, universities and rural health offices typically administer the programs, but 
this is not always the case. In a few states, state medical associations, Area Health 
Education Centers (AHECs) and various units within health departments oversee the 
programs. There is not one central, unquestionably complete directory of these state 
pro grams available to students who want to apply for these loans and scholarships. 

In each of the 47 states, there are one to six different scholarship and loan repayment 
programs. Most states have multiple programs with varying organizational structures. Some 
of these programs are administered by one central state office while, in other states, each 
program is run individually by different offices. Eligibility requirements, the size of the 
programs and various other program characteristics also differ. 

Though funding can range from state-only funds to money collected from private 
foundations, state agencies typically administer these programs and provide much of the 
funding. Over one-third (41 of the 111 programs) receive federal funding from the 
National Health Service Corps. The NHSC provides federal funding to 12 Community 
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Scholarship Programs and 29 State Loan Repayment Programs which are also funded by 
matching state funds. For the NHSC community scholarship program, states and 
communities must provide 60 percent of the funding. States must provide 50 percent of the 
funds under the NHSC loan repayment program. 





Table 2 


Characteristics of State Scholarship and Loan Repayment Programs 


Sources of Funding 


State only; combination of state, federal, or local; 
nonprofits; licensure fees; private foundations; trust fund 
from student fees; unclaimed property 


Funding from NHSC 


41 programs 


Administrative Offices 


Universities, higher education boards, rural health 
offices; state medical associations, AHECs; various units 
within health departments 


Size of Programs 


Varies 



3. Most state legislatures do not require scholarship and loan repayment programs to be 
evaluated. 

Though state legislatures have typically created these programs in state legislation and 
appropriate money to them on an annual or bi-annual basis, relatively few states have built 
accountability measures into these programs. 

Less than one out of five programs (20 out of 111) 
have some kind of report that documents its 
effectiveness. (See Appendix l-C for a table outlining 
the programs and whether they have been evaluated.) 

However, roughly another 20 percent are planning a 
program evaluation in the near future. Though some 
of these administrators did not specify an exact date 
of completion, most did indicate that their evaluations 
would be completed within the next 12 months. 

In contrast to most states, the federal government 
requires scholarship and loan repayment programs 
that receive federal funds to be routinely evaluated. 

The NHSC requires the 41 programs that receive 
federal funds to file quarterly progress reports, as well 
as annual grant applications, as a condition of 
receiving money. Seven of these programs file 
quarterly progress reports with the NHSC and 
conduct additional evaluations either by their own 
accord or by state requirement or request. 

The NHSC is required, by law, to submit this information annually to the U.S. Congress. 
The reporting requirements include the following data elements: 



Figure 1 



Percentage of State Scholarship and Loan 
Programs with Evaluations 




Q Evaluation g Planning EvaJ. □ No Evaluation 
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• Information on rural and urban federally designated HPSA placements; 

• Number and amount of grants to community organizations or participants by year; 

• Number of scholarships or loans by discipline; 

• Participants' educational institution; 

• Length of contract; and 

• Breaches of contracts and the basis of decision for those who have breached contracts. 

This data is reported in the aggregate, instead of on a program-by-program basis. Little of 
this information is believed to be shared with state legislatures. 

Though the NHSC collects a rich body of data from these programs, it does not require the 
programs to collect data on provider retention. The programs that receive NHSC funding 
are not required to track the providers after their service obligation ends to determine if and 
how long they practice in the HPSA. 

4. About three-fifths of the programs either have not conducted or are not planning to 
conduct an evaluation in the near future. Administrators report that, for the most part, 
these programs are either too new or lack the resources — time, staff or money — to 
conduct such evaluations. 

The most common reason reported by program administrators for not conducting a 
meaningful evaluation was that their programs had not been in operation for a sufficient 
period of time. A significant indication of a program's success is its provider retention rates. 
Retention rates are a measurement of providers' length of practice in a HPSA. 

Figure 2 It can take at least 10 years 

to collect accurate re- 
tention rates on physician 
participants given that 
obligations often com- 
mence with their accept- 
ance of educational loans 
during their training period 
and end with a practice 
obligation typically two to 
four years in length. (As 
noted, three-fourths of the 
programs have been in 
existence for 10 years or 
less.) For example, a 
program administrator first 
must wait for the recipients 
to matriculate from med- 
ical school. Then, 

recipients must finish their 
residencies and, finally, 
their service obligation in a HPSA before the state can begin to determine if and for how 
long the providers remained in a HPSA beyond their obligation. 

The second most common reason reported for not conducting an evaluation was a lack of 
resources — a lack of time and staff, a deficiency of funds, or a combination of both. After 
corresponding with program administrators across the country, researchers found that 
programs generally are run by only one or two people who "wear many different hats." 
They often have other responsibilities in addition to running the scholarship and loan 



Figure 2 



Reasons for Not Conducting Evaluations 













H # of Programs 
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repayment programs which vary according to where they work. For example, the duties of 
rural health officials vary from those who staff higher education boards. In addition, many 
of the programs run on "shoe string" budgets. Most of the money goes toward student 
scholarships and loans leaving very little for administrative support. 

Some less common responses for not conducting an evaluation include the following: 

• Several administrators keep statistics on their programs' recipients but have not 
published these data in any kind of report; 

• Seven programs have too few participants to warrant conducting an evaluation; 

• Funding for one program was recently discontinued. As a result, . one program 
administrator plans to conduct an evaluation in the hopes that it will demonstrate the 
program's effectiveness and funding will be reinstated; and 

• Another state official who administers five programs reported that the state is examining 
the programs to determine if they are meeting the state goals. However, she is not 
planning to collect any data to measure the programs' effectiveness. 

5. The evaluations vary greatly in their level of complexity, length and presentation. 

Researchers identified 15 states with 20 evaluated programs. These states include: Arizona, 
Arkansas, Delaware, Georgia, Idaho, Illinois, Michigan, Nebraska, New Jersey, New York, 
Oklahoma, Tennessee, Texas, West Virginia and Wisconsin. Researchers were unable to 
collect evaluations from two of the states — Georgia (two programs) and New York (one 
program). Thus, researchers collected evaluations on 17 programs in 13 states that had 
conducted evaluations. 



Table 3 

Evaluation Characteristics and Intended Audiences 


Evaluation Report 


State/(number of Programs Evaluated) 


Informal document of survey or study 
results department or both 


Idaho (1), Michigan (1), Oklahoma (3), 
Texas (1), West Virginia (1), Wisconsin (1) 


Formal report to state legislature, health 


Arkansas (1), Illinois (1), New Jersey (1), 
Tennessee (1) 


Formal broad rural health report to 
the governor and state legislature 


Texas (2) 


Program Authorization Review to 
the governor and state legislature 


Arizonia (1 ) 


Primary Care Committee report to a 
state legislative health care committee 


Delaware (1) 


A formal report for strictly internal purposes 


Nebraska (1 ) 



The complexity of these evaluations ranged from documenting just a few data elements to 
producing formal reports that were the result of months of work. The length of evaluations 
ranged from one page to more than 100. About half of the documents were very informal, 
simply presenting the data (eight programs), while the other half of the program reports were 
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more formal and produced for distribution (nine programs). In addition, they were written 
for different audiences that include state legislatures, governors, health departments and the 
administrator's office, or some combination of these audiences. 



Recommendations from Program Evaluations 

In addition to collecting evaluations, researchers also interviewed program administrators 
about the impetus for, the process of, and the outcomes from the evaluations. Based on 
advice from program administrators and findings from this study, the following 
recommendations are offered. (See Appendix l-D for write-up of these interviews.) 

1. State legislatures or health departments should require administrators to conduct 
evaluations to ensure program accountability. 

The majority of programs — 13 out of the 17 programs surveyed — conducted evaluations 
because they were required to either by legislative mandates or by requirements or requests 
of health departments. Two of the four remaining programs performed evaluations because 
they had access to student interns who helped conduct the evaluations. One evaluation 
was requested by a university vice chancellor, and the other evaluation was conducted 
because of the program administrator's professional conviction about the issue. 

Given tight state budgets, state officials need to know that they are appropriating money to 
programs that are working. Without annual evaluations, it is difficult to determine if these 
programs are alleviating health care needs in the greatest number of shortage areas. 

A 1994 case study by the University of North Dakota Rural Health Research Center came to 
a similar conclusion. By examining scholarship and loan programs enacted before 1981 in 
10 states, the author found that only four of the 10 state programs monitored retention rates. 
The author recommended that more states research retention issues and develop rural 
health profession policy that gives equal treatment to both recruitment and retention. 

2. A program is never "too new" to be evaluated. 

A common misconception is that a program must be operational for many years before 
meaningful data can be collected. About one-third of all program administrators reported 
that they had not conducted an evaluation because their programs were too new since it 
often takes programs at least 10 years before retention data can be collected . 

In contrast, most of the programs that had been evaluated were assessed before they had a 
significant number of providers who had completed their service commitments. More than 
three-fourths of the programs — 13 out of the 17 surveyed — were first evaluated within 10 
years of operation. Though retention rates are important, program evaluations need not be 
restricted by this measurement. 

These "younger" programs collected some of the following data: 

• A listing of students receiving new and continuing scholarships; 

• Where participants were raised and where they plan to be practicing (rural versus urban 
areas); 

• Expenditures and fiscal history of program; 

• Number of scholarships by public and private medical schools; 

• Ethnicity and gender of recipients; 
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• Number of academic failures; and 

• Loan recipients' perceptions of the importance of the program in deciding to practice in 
HPSAs. 



Table 4 

State Scholarship and Loan Repayment Programs by When They Were First Evaluated 


Years Before First Evaluated 


Number of Programs 


1 Year 


7 


2-5 Years 


4 


6- 10 Years 


2 


1 1 + Years 


4 


Information Unavailable 


3 


TOTAL PROGRAMS 


20 



3. Program administrators should examine the data gathered by other state scholarship 
and loan programs and the NHSC to determine what information would be most useful to 
collect. 



Though there is much variance, most scholarship and loan repayment programs attempt to 
collect information on: 

• Provider practice location and specialty; 

• Length of time in obligation; and 

• Retention status in terms of those who default, those who fulfill their obligation, and 
those who stay in the HPSA beyond their obligation period. 

As noted, retention rates often are the goal of state program evaluations. Of the 17 
evaluations that researchers collected, eight of them reported on program participants' 
retention rates, i.e. the number of participants who practiced in the HPSA past their 
obligation period. These state programs include Arkansas, Texas, Tennessee, Oklahoma, 
Nebraska, Illinois and Michigan. The program administrator in Oklahoma collects retention 
data on two programs, whereas retention data are collected on just one program in each of 
the other states. 

Some other data reported in the evaluations include: 

• Where participants were raised and where they plan to be practicing (rural versus urban 
areas); . 

• New programs participants have implemented in their communities; 

• The communities' thoughts on the program's responsiveness to inquiries, and the 
communities' willingness to write to legislators in support of more funding for the 
program; 

• Students receiving new and continuing scholarships; 

• Expenditures and fiscal history of program; 

• Number of scholarships by public and private medical schools; 
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• Ethnicity and gender of recipients; 

• Type of high school of those serving beyond obligation; 

• Academic failures; 

• Monetary repayment; 

9 Number of buyouts by school; 

• Number of recipients still practicing in state; 

• Comparisons of location of residency to retention in obligated service areas; 

• Practice sites; 

• Patient and practitioners' satisfaction with the program; 

• Provider sensitivity to community needs; 

• Recipients' previous exposure to underserved area and other factors influencing 
practice in such settings; 

• Amount of loan; and 

• Current patient workload. 

In addition to looking at program evaluations from other states, program administrators 
should examine some of the reporting requirements of state community scholarship and 
loan repayment programs that receive funds from the NHSC. The reporting requirements 
are contained in four separate sections of the current statute and are reported annually to 
the U.S. Congress. (See Appendix l-E for NHSC's reporting requirements.) The NHSC's 
reporting requirements can serve as a baseline for some uniform data collection efforts for 
those programs not receiving federal money. 

4. Evaluations can be conducted effectively with limited resources. 

The cost and personnel needed to conduct evaluations depends on whether they are 
conducted internally or by an independent entity. Most program administrators — 15 out of 
1 7 surveyed— conducted their evaluations using in-house resources and staff. One program 
was reviewed by a legislative task force, and another hired an independent research firm to 
conduct the evaluation. 

Of those evaluations conducted in-house, three-fourths of them evaluated their own 
programs within one day to two weeks. For the most part, administrators surveyed program 
participants by phone or mail and then entered the data into a data base or spreadsheet. 
Only two programs used already existing data bases to collect data on their recipients. 
These evaluations were conducted economically because administrators were able to 
simply update the information from the previous year's report. In most cases, program 
administrators were even unable to determine the costs associated with the evaluations 
because they used internal resources. 

For those administrators who either used independent consulting firms or planned to use 
one, the costs and time were much greater. For example, New Jersey's program contracted 
with an outside firm to conduct its evaluation which was completed in eight months at a 
cost of $10,000. Tennessee's program plans to contract with a state university for $30,000 
to perform an evaluation of its program directed at physician assistants and advanced 
practice registered nurses. 

5. Programs should be evaluated on a routine basis. 

It is crucial for program administrators to recommend improvements to the program based 
on good, up-to-date information. Annual surveys will not only record a rich history of the 
program but also will ensure that the information is current. About half — 9 out of 17 — of 
the programs are evaluated on a regular basis. Six are evaluated annually, two bi-annually 
and one on an ongoing basis. The other half of the programs are evaluated either subject to 
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the availability of funds or on an irregular or multi-year basis. The program in Delaware 
was evaluated on a one time basis by a legislative task force. 

6. Evaluations have proved to be valuable. 

Nearly all program administrators — 16 out of 17— reported that the evaluations were very 
useful. These administrators cited that their program evaluations led to the following 
outcomes: 

• A justification of the programs' existence and demonstration of their effectiveness; 

• An increase in program funding, 

• Anecdotes to state legislators on the programs' value, 

• Recruitment of new participants by providing data on prospective sites, 

• Recommendations for improving and restructuring the programs; 

• An expansion of programs' roles and focus; 

• A move and consolidation of administrative offices; and 

° Establishment of a trend line to compare and contrast program characteristics across 
time. 

7. Evaluation efforts of programs in two states — Illinois and New Jersey — serve as 
innovative models. 

The scholarship and loan repayment programs in these states share a common mission — to 
increase the availability of primary care services in underserved areas. These programs 
were also required to be evaluated to determine whether this mission is being met. 
Otherwise, these programs and their evaluation processes have very little in common. 
Thus, evaluations of these two state initiatives serve as models to demonstrate the great 
variance in program evaluation and to highlight the strengths of the different types of 
evaluations. (See Appendix l-F for more details.) 

Lllinois 2 was chosen for collecting consistent, extensive data, including comprehensive 
retention data, over the past 1 1 years. The program administrator collects retention data by 
the number of providers who practice beyond the requirement, the number of years served 
beyond the requirement and the practice location and type of high school of those serving 
beyond obligation. 

New lersey 3 on the other hand, was chosen for investing the resources to hire an 
independent, unbiased research firm that collected comprehensive information through site 
visits, interviews with program participants and members of the communities and 
discussions with program staff and state agency officials. Though these programs were 
evaluated quite differently, both evaluations led to or are in the process of leading to 
improvements in the program and justification for state funding. 

8. In evaluating program effectiveness, state administrators and legislators need greater 
opportunities to learn about other state scholarship and loan repayment programs. 

Although state legislators are responsible for appropriating funds to the majority of state 
scholarship and loan repayment programs, many of them have not had the opportunity to 
learn about the successes, pitfalls and merits of their own programs as well as those of other 
states. State legislators could benefit from educational opportunities about these programs 
and other medical education programs in the state aimed at alleviating HPSAs. 
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Table 5 




Comparisons of Program Evaluations in Illinois and New Jersey 


Categories 


Illinois 


New Jersey 


Program Name(s) 


Medical Student Scholarship Program 


Primary Care Physician and Dentist 
Loan Redemption Program 


Year Program Became 
Operational 


1978 


1992 


First Evaluation 


1984-1985 academic year 


Program in existence for two years 
when first evaluated. 


Reason for 

Conducting 

Evaluation 


Legislative mandate 


Health department evaluated many 
programs, including theirs. 


Information Collected 


Students receiving new and continuing 
scholarships; expenditures and fiscal 
history of program; number of 
scholarships by public and private 
medical schools; ethnicity and gender 
of recipients; retention rates; recipients' 
medical specialty and geographic 
distribution; recipients' practice 
location; type of high school of those 
serving beyond obligation; academic 
failures; monetary repayment; and 
number of buyouts by school. 


Recruitment; best practices from 
programs in Connecticut, Florida, 
Louisiana, Massachusetts, Maine, 
Maryland, New York, Texas and 
Washington; practice sites; patient 
and practitioners' satisfaction with the 
program; provider sensitivity to 
community needs. 


System of Data 
Collection 


Use Data Ease software and mail 
surveys every six months on prepared 
forms to track students, residents and 
those in practice. 


Used outside research firm, MRH 
Evaluations, to conduct study. 


Cost and Personnel 
Time 

Frequency of Updates 


Two weeks; $$ unknown 
Annually — every March 


Roughly $10,000; Eight months 
Subject to the availability of funds. 


Outcomes 


The evaluation has given us 
confidence in that we can demonstrate 
the effectiveness of the program with 
hard numbers. 


Recommendations from the 
evaluation were incorporated into a 
proposal to the governor. 
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WAMI Conference 



The annual WAMI conference may serve as a model for designing regional conferences in 
which to educate legislators about the HPSAs in their states and efforts to eliminate them. 
WAMI, an acronym denoting the states: Washington, Alaska, Montana and Idaho, is a 
regional education program of the University of Washington School of Medicine. Medical 
students from these states, plus the newcomer, Wyoming, attend the University of 
Washington medical school, but spend their first year at an in-state university. The 
curriculum at each site is similar to and compatible with the University of Washington 
curriculum. 

For at least 15 years, WAMI has held annual conferences to promote group discussions 
among legislative and other participants to explore opportunities for future growth and the 
educational direction of WAMI. This year, 145 people attended the day and a half 
conference, 65 of which were state legislators (roughly 13 to 15 legislators per state). 
Participants for whom all expenses are paid are chosen by program administrators in each 
of the five Northwest states. During the conference, participants are updated on WAMI, 
and individual state meetings are held. 

A half day is devoted to open discussion from conference participants in order for the 
legislators to truly understand the program's impact on their state. Each participant receives 
a notebook with state specific evaluations that include information on the number of health 
care providers returning to their home state, the number of out-of-state residents doing 
clinical practice in state, the cultural background of students, the geographic location of 
students' hometowns and the number of graduates in rural practice. General information 
on the University of Washington medical school, as well as contact names and phone 
numbers, are also included in the conference packets. 

The success of this legislative conference can be a model for promoting informed decision 
making in state legislatures about medically underserved areas. State legislators who hold 
the purse strings to these programs must understand them and their evaluations in order to 
enact legislation to eliminate, enlarge or modify these programs when necessary. 

Program administrators could also benefit from such conferences. Administrators seem to 
know other program administrators in their own state, but have little or no contact with 
those outside of their state. A recent conference of state and NHSC officials was an 
important occasion to establish a dialog regarding the positive and negative aspects of the 
programs and report on program outcomes, including recommendations on how to enhance 
the programs to ensure continued success. This type of meeting, or perhaps smaller 
regional meetings, should be replicated to give other program administrators a chance to 
talk with each other. 

When asked about this idea, nearly all administrators saw a multi-state meeting as a great 
opportunity to develop a network and learn from other program administrators' experiences, 
as well as lessons that they have learned from conducting evaluations. The biggest 
obstacle, however, is travel funds. Most of the administrators have little or no travel funds 
available. Thus, travel scholarships would have to be made available. 
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9. Program administrators have learned important lessons based on their program 
evaluations. 

Researchers asked program administrators what advice they would give other administrators 
based on their program evaluations. Though much of this information has already been 
provided, below is some further advice: 

• Think about who wants to know what and what information is useful before you 
evaluate the program. 

• Put in place a user-friendly, manageable data base to track students, so anyone can use 
it. Include a comments section in the evaluation. Over time, many of these programs 
may be managed by different program administrators. 

® Talk to program participants on a regular basis. Although phone surveys are more 
costly and time consuming, participant interviews are useful for gauging the feelings of 
the program's influence on training and practice locations. Those program 
administrators who conducted phone interviews stressed the importance of talking to 
program participants, as well as site coordinators in the communities being served. 

• Keep surveys simple and consistent to ensure that results are easily comparable with 
previous and future evaluations. Keep good records throughout the years, so the 
evaluation can be easily updated. 

• Having students conduct the evaluations is one of the most cost-effective options. This 
option is viable particularly for those programs located at universities. 
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Rationale 

Traditionally, state support for medical education takes the form of some or all of the 
following: 1) operating subsidies to teaching hospitals and clinics; 2) direct support of 
clinical education programs such as residencies, internships and preceptorships (and of 
Area Health Education Centers in some states); and 3) Medicaid reimbursement to hospitals 
for certain teaching costs. Appropriations are often not separately identified, and several 
states have found it difficult to isolate service reimbursement from clinical education 
payments under Medicaid. Many states provide specific funds for graduate training in 
family medicine and other primary care specialties. In recent years, at least six states have 
appropriated more than $5 million annually for these programs. This strategy is often seen 
by legislators as solving problems of rural access to physicians. States annually provide 
nearly $3 billion in appropriations and $1 billion to 2 billion in Medicaid payments to 
health professions training programs and teaching hospitals. 

Educating physicians for primary care practice has recently become a high priority in many 
medical schools. Several states have been major supporters of this trend by placing greater 
pressure on state-supported schools to train more generalists. These states are modifying 
various forms of educational support and financial incentives for students and residents in 
the hope of shifting the balance between primary care and specialty practice. 

These changes come at a time when states have been forced to scrutinize their support for 
medical education and teaching hospitals. In the late 1980s, most states began 
experiencing major fiscal problems. State dollars as a proportion of medical school budgets 
have declined, despite a near doubling in the past 15 years in the amount of state funds that 
the institutions receive. Some states, perceiving an oversupply of physicians, reduced their 
support for medical education. During this period, medical schools became increasingly 
dependent on patient care revenues. 

Also, state Medicaid programs have rarely agreed to provide reimbursement for the 
additional costs of teaching in ambulatory sites. For most ambulatory education programs 
that train primary care residents, care is provided to large numbers of Medicaid and 
indigent patients. Typically, these sites earn no additional revenues from Medicaid to cover 
teaching costs, which makes it difficult for many programs (e.g., HMOs) to be competitive. 
At the same time, states have had mounting concerns about the maldistribution of primary 
care physicians and the unmet needs of many rural and inner-city areas. Physician 
shortages in these communities persist and, in some areas, have worsened. 
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Increased efforts by states to pressure medical schools and teaching hospitals to train more 
generalist physicians are designed to: a) achieve some congruence between the public need 
and existing supply of physicians, and b) more carefully account for all state contributions 
to medical education. To this end, states have implemented or are considering 
implementing the following strategies: 

• Establishing family practice training programs. At least 13 states have passed 
legislation that specifically encourages or mandates the creation of departments of 
family medicine or other family practice training programs in state-supported schools. 

• Targeted appropriations. More than 40 states have created special grant programs for 
family physician training, and about half of the states specify appropriations for family 
practice education. The amount and scope of these appropriations continue to wax and 
wane. 

• Outcome-based measures. A half-dozen states have enacted laws linking education 
funding to specific and measurable outcomes focusing on the specialty mix of graduates 
and residents trained. Typically, schools in these states are required to prepare a plan 
with the goal of training a large proportion (typically 50 percent) of their graduates in 
primary care by a certain date without additional state funds. 

• Reforming curricula and emphasizing community-based education. States are 
increasing the number of required and elective clerkships, rotations and other clinical 
training arrangements, typically in community-based settings, for generalist-minded 
medical students and residents. The Texas Legislature requires all third-year medical 
students to complete a clerkship in family medicine. 

• Preferential admissions and early intervention in secondary schools to encourage 
health careers for minority students and students from underserved communities. 

® Medicaid payment under managed care for graduate medical education. Despite 
budget pressures, a few states adopting Medicaid managed care programs have 
maintained and in some cases improved the integrity of CME support by channeling 
such funds directly to teaching institutions. Tennessee has opted to pay the medical 
schools directly thereby encouraging greater training of primary care professionals in 
community-based settings. Other states are considering reorienting their Medicaid 
GME payments toward primary care as they implement managed care. 

• Create or expand support for advanced practice nursing and physician assistant 
training. Currently, just a handful of nurse practitioner and physician assistant training 
programs receive any significant state funding for operational support. In general, just 
half of all NP programs receive any grant support beyond student tuition payments. 
Many of these programs are struggling to attract faculty, establish training sites and 
remain financially viable. 

• Create or increase stipends for primary care residents and preceptors. As many 
generalist training programs attempt to educate more students and residents in 
community-based settings with few resources, there is often an increased reliance on 
the use of community preceptors. Often, these providers are asked to volunteer their 
time. 

• Earmarking practice plan and tuition revenues for generalist training. At least one 
state, Kansas, mandated an outside study to look at the efficiency of the state medical 
school's practice plans and their relationship to the teaching hospital. Many of the 
primary care training programs in the state, as elsewhere, are experiencing significant 
financial difficulty and could benefit from a reallocation of plan revenues generated 
mainly from specialty departments. 
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Purpose and Methodology 



Several states have enacted legislation in recent years instituting one or more of the above 
strategies to reform medical education. (See Appendix ll-A.) Between 1985 and 1993, 34 
states passed laws requiring various institutions to implement these strategies. The most 
commonly enacted strategies were to create or expand community-based family practice 
residencies and otherwise to expand community-based clinical training in medically 
underserved areas. 

In the summer of 1996, various officials in these 34 states were surveyed by mail to 
determine whether, and to what extent, any or all strategies enacted in their state had been 
evaluated. Even though many of these strategies had only been enacted within the past five 
years, it was hoped that at least a few of the states had attempted to evaluate early 
performance. Given the low number of states expected to have performed evaluations and 
provide such information, summaries of a select number of these state evaluations will be 
presented as case studies. 

T his study has adopted a broad definition of evaluation, accepting any written effort to 
document or analyze significant trends and progress in a program's implementation . While 
some individual training institutions have conducted evaluations of their own efforts to 
reform medical education, this study is mainly interested in reviewing and describing 
evaluations of multi-institutional or state-directed initiatives to improve medical education. 
It is thought that such evaluations are likely to harbor less bias toward a particular training 
program or institution's interest or mission and be viewed more favorably by state legislators 
and other health officials. 



Findings 



To date, most medical education reforms enacted bv state legislatures have not been 
evaluated . Of the 34 states surveyed, various officials from 10 states provided copies of 
acceptable evaluations performed of recent state-directed or multi-institutional initiatives to 
reform medical education. 

The most common strategy that was the subject of the evaluations was state support for 
community-based family practice residencies . (See Appendix ll-B.) The most frequently 
collected types of information used to perform the evaluations was trend data on student 
interest in family medicine; graduate totals in family medicine in relation to current supply 
and need; graduate placement and retention rates in underserved communities; and state 
residency program funding. 

Most of these evaluations were performed very recently and thus little is known about 
whether they have had any impact on medical education funding or other program 
changes. 



Summaries of Evaluations in Four States 



The content and structure of these evaluations are very different and the information 
collected for them does not easily allow for the tabulation of simple comparisons. 



O 



National Conference of State Legislatures 



25 



Evaluation of State Efforts To Improve the Primary Care Workforce 



18 



Therefore, specific findings associated with each state's evaluation are presented in a case 
study format. Evaluations performed of medical education reforms in four states (Illinois, 
Kentucky, South Dakota and Texas) were selected because these cases either reflect a 
history of strong state interest in the success of medical education reform or the evaluations 
themselves, although not academically rigorous, are well focused, clearly interpretable, and 
perhaps of greater pragmatic benefit to interested state legislators. 



Illinois 

Rural/Downstate Health Act 

The Illinois legislature began to address the need for new primary care training programs as 
early as 1985. That year a law created rural rotations in which medical residents would be 
allowed to receive some training in a rural setting. In 1992, the legislature amended the 
Rural/Downstate Health Act to require the Center for Rural Health to cooperate with the 
University of Illinois' efforts to address health care needs of downstate residents. Such 
cooperative efforts included developing innovative educational strategies to graduate 
primary care physicians. In requiring the university to expand its efforts to enroll and train 
more primary care physicians, the law obligates both the undergraduate and graduate 
primary care training programs to increase education and service initiatives at satellite sites 
in rural underserved and health professional shortage areas. 

A resolution that passed the legislature in 1993 called for the creation of a special joint task 
force to analyze family physician shortages in the state. The task force was required to 
review the state's family practice training programs and to make recommendations to 
increase funding of these programs and the number of family physicians trained. The task 
force's report to the governor and legislature was completed in early 1994 as part of a larger 
report by the Illinois Health Care Reform Task Force. The report's broad recommendations 
on medical education reform include several suggestions for encouraging more community- 
based training. These include: 

• Ensuring a mechanism is in place to better tie state medical education funds directly to 
the state's needs for health care providers; 

• Ensuring that curricula provide students and residents with a balance between hospital- 
based, subspecialty training and community-based primary care training, with a greater 
proportion shifted to ambulatory settings; 

• Establishing and maintaining a community-based infrastructure that supports medical 
education and meets local needs for health personnel; 

• Expanding AHECs to underserved communities to coordinate and support primary care 
education at the community level; 

• Creating community-based networks to provide training for local providers to serve as 
community faculty; 

• Requiring residents to train at community health centers; 

• Providing direct and indirect GME funding to nonhospital, community-based sites for all 
training-related costs from a state pool to which all payers would contribute. Rural 
training sites should receive preferential funding under Medicaid; and 

• Requiring medical schools to actively develop relationships with rural providers and to 
attempt to match graduates with rural practice sites. 

At least part of the report's recommendations have been considered by the legislature. A 
1994 law further amended the Rural/Downstate Health Act by requiring the Department of 
Public Health to establish a service-education program to improve the supply of primary 
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care professionals in underserved areas. The program involves the state's area health 
education centers. 

Studies on Medical Education 

In 1995, the legislature created the Primary Care Medical Education Advisory Committee. 
The committee is charged with coordinating and evaluating the activities of three distinct 
state agencies that fund medical education in the state and advising the legislature about the 
most appropriate ways to distribute state medical education funds. Subsequent to, but 
separate from, this decision, one of three state agencies — the Department of Public Aid — 
made the decision to discontinue support of graduate medical education through Medicaid 
due to a budgetary crisis, and the $200 million it provided to teaching hospitals in 1 995 was 
not available in 1996. (Illinois is only one of two states that currently do not support 
graduate medical education under Medicaid.) In early 1996, the Illinois Academy of Family 
Physicians issued a report evaluating state efforts to finance graduate medical education and 
recommended that the Advisory Committee address several reforms the state should 
institute. The Advisory Committee continues to meet, but as yet has not produced a report 
detailing any evaluations or recommendations. 

Also in 1996, the University of Illinois at Chicago released the results of a comprehensive 
survey of primary care residency programs in the state. The study was funded by the Illinois 
Board of Higher Education and the Chicago Community Trust. The survey found that the 
majority of physicians who complete primary care GME programs in the state plan to 
practice primary care and about half of these remain in Illinois to practice. The GME 
programs that most successfully recruit state medical school graduates are family practice 
and obstetrics/gynecology. As is the case elsewhere, the study found that only about 15 
percent of training time for residents is spent in physicians' offices (except in family practice 
where it is 35 percent). There is very little use of community-based training sites. 

Kentucky 

In 1990, omnibus health reform legislation in Kentucky established a rural family practice 
residency program and created the University of Kentucky Center for Rural Health, whose 
mission is to increase the number of health professionals practicing in underserved eastern 
Kentucky. More recently, as part of another major health reform bill passed in 1994, the 
legislature ordered the state's two public universities to jointly establish and operate at least 
six community-based family practice residency programs, one in each of the state's 
congressional districts. The initiative creates residency slots in community-based sites to 
accommodate all medical school graduates entering family practice. Participants receive 
scholarships and, later, stipends and must fulfill a service obligation period of one year in an 
underserved area for every one year of loan forgiveness. Because many of these new 
residency programs are to be located in very rural communities, it is expected it will be 
difficult to recruit both residents and faculty. The state is considering instituting preferred 
resident stipends and salaries for these sites. 

The Center for Rural Health has begun to track and evaluate its progress in graduating 
primary care professionals and retaining them in eastern Kentucky and other underserved 
rural areas. To date, of the total graduates in family medicine, nursing and allied health, 76 
percent have remained in rural Kentucky to practice and another 3 percent are in rural 
counties in other states. Of the 216 students admitted to training since 1991, 68 percent are 
from rural eastern Kentucky and just 9 percent are from outside the state. 
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A 1994 study of retention patterns of primary care physicians in eastern Kentucky asked a 
sample of these providers to rank what items were most important to retention. Those items 
rated the most important were availability of relief coverage, quality of local schools, 
compatibility with the medical community, availability of quality housing, consultation with 
a specialist via telephone, and availability of practice partners. There was no specific 
mention of the resources of the Center for Rural Health as a retention factor. 

South Dakota 



State funding support for the state's two urban-based family practice residency programs 
was very controversial in 1995. Conflict existed as to whether it is feasible and more 
effective for these programs to be based in rural underserved community settings. There 
was and continues to be a growing concern about the effectiveness of state-sponsored 
recruitment and retention programs and their competitiveness with similar programs in 
neighboring states as the number of full-time family physician practice sites continues to 
decrease in South Dakota. 

In 1995, the governor recommended an appropriation of less than half of the amount 
requested by the residencies. In 1996, the legislature enacted a measure that compromised 
on a $675,000 appropriation. The law also required a study of residency program funding 
issues involving the medical school, residency programs and the Department of Health. 

Charged with conducting this study, the Governor's Task Force on Improving Recruitment 
and Retention of Family Physicians made its findings and recommendations available in 
November 1996. The task force was to determine the appropriate level of state funding for 
the family practice residency programs, establish a stable funding mechanism, and 
determine how to maximize state funds to provide incentive for the placement of residency 
program graduates into rural areas. 

After extensive analysis and discussion, the task force recommended that stable funding be 
achieved by maintaining the 1996 base funding of $675,000 with annual increases. In 
addition, it was recommended that an advisory committee be created to determine ways to 
get more state medical school graduates into South Dakota family practice residencies. 
(Eight of 10 state medical school graduates who choose in-state family practice residencies 
will remain in the state to practice.) The advisory Committee would also assist the 
Department of Health in determining how to maximize state funds and provide incentives 
for the placement of providers in rural areas. The task force believed the formation of an 
advisory committee would ensure that independent professionals continue to review and 
evaluate on a regular basis those residency training programs that receive state funding. 

Texas 

For many years, state officials have been concerned about the overall lack of primary care 
providers and their maldistribution in Texas. Texas ranks below the national average in the 
number of generalist physicians practicing per 100,000 population and has a higher percent 
of its population underserved (1 1.2 percent) than the country as a whole (10.5 percent). 

State Legislation Addressing the Primary Care Workforce 

In 1987, Texas lawmakers passed a law creating a nine-member "Special Task Force on 
Rural Health Care Delivery in Texas." The charge of the task force was to: 1) define 
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minimal desired medical care for rural counties and communities, taking into consideration 
population, geography, proximity to tertiary care centers, physician manpower and 
transportation availability; 2) define the resources available and/or needed to provide a 
voluntary plan to meet the needs of the state's counties, including methods of financing the 
implementation and operation of such a plan; and 3) seek consensus among affected parties 
to support the plan. 

The final report of the Task Force was issued in 1989. In response, the legislature enacted 
an extensive law in 1989 relating to rural health. The 1989 measure required the Texas 
Higher Education Coordinating Board (THECB), the newly established Center for Rural 
Health Initiatives, medical and other health care education schools to cooperate to improve 
and expand programs for rural areas, including the following: 1) encourage and coordinate 
the creation or expansion of a rural preceptor program among medical schools and 
teaching hospitals; 2) require family practice residency programs to provide an opportunity 
for residents to have a one-month rotation through a rural setting; 3) develop relief service 
programs for rural physicians to facilitate access to continuing medical education; and 4) 
require medical schools to incorporate a third-year clerkship in family practice for all 
medical students and report on its efforts to fulfill the intent of having at least 25 percent of 
first year primary care residents in family practice. Until recently, Texas was the only state 
whose legislature required all third-year medical students to complete a clerkship in family 
medicine. 

A 1995 law approved several new measures to improve the supply of family practice 
physicians. Among the provisions pertaining to medical education, it established new 
statewide preceptorship programs in general internal medicine and general pediatrics 
modeled after the existing family practice preceptorship program; provided an additional $1 
million for a family practice residency training program (the first increase in state funds for 
the program since 1988); established three family practice residencies to provide services in 
economically depressed or rural areas of the state; and provided support for additional 150 
community-based primary care residency positions phased in over five years, although per- 
resident allotments will not increase. 

Undergraduate Medical Education Production Trends 

In 1995, Texas' eight medical schools graduated 1,163 physicians, up from 1,013 in 1981. 
According to a survey of these 1995 graduates by the Texas Medical Association (TMA): 

• percent of respondents chose to do their postgraduate training in Texas. This is a major 
reversal from the 1980s when 60 percent of Texas medical graduates left the state for 
postgraduate education because Texas did not have enough first-year residency 
positions to meet demand; 

• Fifty-seven percent of respondents chose postgraduate study in the generalist 
specialties: family practice, internal medicine, pediatrics and obstetrics-gynecology. 
Forty-eight percent of respondents indicated they intended to practice in these 
generalist fields upon completion of training; and 

• Thirty-seven percent of respondents completed a preclinical family practice 
preceptorship; nearly half of these identified family practice as their postgraduate 
specialty. Students completing family practice preceptorships were six times more likely 
to choose family practice as their specialty than were students who did not participate 
in a family practice preceptorship. 
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Graduate Medical Education Production Trends 

In 1995, Texas' 414 allopathic and 29 osteopathic medical residencies graduated about 
1 ,1 00 residents and fellows. According to a survey of these graduates by TMA: 

• Of the 544 respondents who chose to practice a medical specialty, more than half (55 
percent) trained in the generalist fields of internal medicine (21 percent), pediatrics (15 
percent) and family practice (1 1 percent), and in obstetrics-gynecology (8 percent). 

• More than half (56 percent) of the respondents spent the majority of their precollege 
years in a large city or suburb; just 20 percent lived in a small city or rural community. 
Only 12 percent of respondents said they intended to practice in a small city or rural 
community. 

• In 1994, 61 percent of all first-year residents in Texas' graduate training programs were 
in primary care specialties. 

Primary Care and Undergraduate Medical Education 

The 1989 law mandated that third-year family practice clerkships be instituted starting with 
the 1990-1991 school year in each of the eight medical schools. As of late 1994: 

© All eight schools report compliance with the family practice clerkship requirement and 
all schools increased the number of primary care faculty; and 

• All schools report increasing student exposure to community-based primary care earlier 
in the curriculum, including offering preceptorship experiences in family practice for 
students after their first year. Each school relies on community physicians to supervise 
medical students in preceptorships and required or elective clerkships. 

About 1,100 students annually spend one month in a rural clerkship. No additional funding 
was provided to the schools to implement this measure. Each school is directed to expend 
specific amounts from their state appropriations or institutional funds for these clerkships. 
For 1996-97, the legislature mandated a total of $7.3 million be spent on these clerkships 
across eight medical schools. 

However, it is not clear whether the clerkship requirement is a significant influence on a 
student's decision to go into family medicine. A 1993 report by the Office of the State 
Auditor of the family practice clerkship indicated it is too early to determine the 
effectiveness of the clerkship in increasing the number of family practice physicians. 

Primary Care and Graduate Medical Education 

In 1977, the legislature first made state financial support available for postgraduate training 
in family medicine. House Bill 282 gives the Texas Family Practice Residency Program, 
administered by the THECB, authority to allocate state funds to family practice residencies 
on a contract basis. The program initially in 1977-78 appropriated about $852,000 to 12 
operating residencies to support 267 positions and to nine new programs for planning 
activity. By 1994-95, the state provided about $8 million to 23 programs sponsored by 
Texas medical schools supporting more than 460 positions. Currently, there are 25 state- 
funded programs supporting 698 positions. (Another six family practice residency programs 
and 1 00 positions do not currently receive state support.) 
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Effectiveness of State Support 

The rural rotation program, required by the 1989 law and begun in 1990, started with two 
family practice residents the first year, eight the second, and interest in this type of 
residency continues to increase. In 1996, 55 residents were expected to participate. 
Currently, family practice residents may select a supervisor from more than 100 volunteer 
rural family physicians statewide. 

The impact of the rural rotation requirements has been beneficial — both because rural 
practice was incorporated into the core curriculum for medical students and it was elevated 
to the level of an optional rotation in residency programs. Consequently, there are 
increased opportunities to expose more physicians in training to rural practice. 

Currently, at least 20 percent of medical school graduates go into a rural county to practice. 
A recent retention study by the THECB found that as many as 66 percent of the graduates of 
state residency programs between 1972 and 1983 remained in the area where they 
completed their residencies. In general, nearly 90 percent of the more than 2,000 family 
physicians trained in state-funded residencies have remained in the state to practice. Of 
those, 40 percent work in towns of 50,000 people or fewer. Furthermore, family practice 
residencies in 1994 reported providing more than $60 million in direct patient care service, 
$25 million of which was uncompensated care. 

Yet, particular regions of the state remain in special need of family physicians. A 1989 
report by the state health department recommended that Texas' medical schools develop or 
expand family practice residencies along the Texas/Mexico border M as their top priority." In 
part because of the 1995 appropriations bill, the state's medical schools and community 
hospitals are in various stages of expanding or creating seven family practice residency 
programs in medically underserved areas of the state. 



Conclusion 

Despite the significant action taken by state legislatures in the past 10 years to create and 
improve loan repayment and medical education initiatives aimed at increasing the supply of 
primary care providers practicing in medically underserved areas, much less attention has 
been placed on ensuring that these initiatives are effectively monitored and evaluated. 
Most laws carry no appropriation to evaluate nor contain measures to enforce a new 
program's effectiveness, thus providing the state minimal evidence of its success. In general, 
few sound evaluations have been performed of these various reform strategies, particularly 
those initiatives common to many states. 

Local needs for primary care providers are expected to persist, the size and scope of 
renewed funding for graduate medical education under Medicare and other federal 
programs remain unclear, and tight state budgets are projected to continue. Therefore, it 
will be incumbent upon state decision makers to demand some measure of accountability 
from programs. Efforts to critically evaluate the success or failure of these programs can be 
of considerable help to state legislators as they set priorities for short- and long-term issues, 
solutions and allocation of funds. States need to increase their evaluation of these primary 
care workforce incentive programs to expand the positive aspects of these programs and 
discontinue those less effective components. 




National Conference of State Legislatures 



24 



Evaluation of State Efforts To Improve the Primary Care Workforce 



In recent years, a growing volume of legislation to improve the primary care workforce, 
particularly to reform medical education, has called for some explicit form of evaluation to 
measure the progress and success of individual programs. This reflects the increasing 
interest of lawmakers in making programs more accountable. Such efforts may also lend 
support to a state's decision to continue or expand a thriving program. It is important as 
well that evaluations be performed routinely to document a program's impact in targeted 
communities. 



1. Gibbens, Brad. Placement and Retention Issues in State Scholarship and Loan Programs. The 
University of North Dakota Rural Health Research Center, University of North Dakota School of 
Medicine. 1994. To order, call (701) 777-3848. 

2. For more information, see Illinois Department of Public Health, Center for Rural Health. Medical 
StudentScholarship Program: Annual Report. 1 995. To order, call (21 7) 782-1 624. 

3. For more information, see Hoppe, Margaret R. and Kindman-Koffler, Bette. Evaluation of the 
Effectiveness of the Primary Care Physician and Dentist Loan Redemption Program of New Jersey. 
May 17, 1994. To order, call (201) 982-4605. 
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Appendix l-A. Scholarship and/or Loan Repayment Survey 



Please complete and return by October 10, 1996. 
Name: 



Organization: 
Address: 



Phone: 



1. Have you or someone else in your state conducted an evaluation of your state's scholarship and/or loan 
repayment program(s)? (You can check more than one box, but please write the program name(s) next to each 
line.) 



□ Yes. Evaluation is enclosed: 

(Specify program name) 

□ We are planning on conducting an evaluation and will send you 

this information in the future. 

(Specify date & program name) 

□ No. We are not planning to conduct an evaluation because: (You can check more than one box; 
please write the program name(s) next to each line.) 

□ Our program is too new, and thus information is unavailable. 

□ We do not have the personnel or time to conduct an evaluation. 

□ We do not have the money to conduct an evaluation. 

□ Other: 



2. In an effort to identify ALL state-supported scholarship and/or loan repayment programs, particularly those that 
have been in operation for several years, please provide information on those programs and contact names (if you 
are not the appropriate person). 

Program: 

Contact: 

Address: : 

Phone: 



BEST COPY AVAILABLE 
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Program: 

Contact: 

Address: 

Phone: 

Program: 

Contact: 

Address: 

Phone: 



Thank you for participating in this study! 



Please return to: 

Wendy Fox-Crage»NCSL»444 N. Capitol St., N.W., Suite 51 5*Washington, D.C. 20001 
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Appendix l-B. Directory of State Scholarship and Loan Repayment Programs 



state 


PROGRAMS 


NHSC/STATE 

FUNDED: 

EVALUATION 

REQUIRED 


CONTACTS 


AL 


Community Scholarship Program 


V 


Cleve E. Money 

AL Dept, of Public Health/Primary Care 
& Rural Health 
434 Monroe Street 
Montgomery, AL 361 30 
(334) 613-5396 




Board of Medical Scholarship Awards 




Freida Baldwin 

Board of Medical Scholarship Awards 
Volker Hall, P1 15 
Birmingham, AL 35294-0019 
(205) 934-4384 


AZ 


AZ Medical Student Loan Program 




Maggie Cumble 

College of Medicine Financial Aid 
University of Arizona 
1501 N. Campbell 
Tucson, AZ 85724 
(602) 626-7145 




AZ Loan Repayment Program 


V 


Belinda Ehlert 

AZ Loan Repayment Program 
1 740 W. Adams, Suite 301 
Phoenix, AZ 850Q7 
(602) 542-1216 


AR 


AR Rural Medical Practice 
Student Loan/Scholarship Program 

Community Match Student Loan & 
Scholarship Program 


X 


Tom South 

College of Medicine Student & 
Academic Affairs 

University of Arkansas for Medical 
Sciences 

Financial Aid - Slot 709 
4301 W. Markham Street 
Little Rock, AR 72205-7199 
(501)686-5813 


CA 


NHSC/CA State Loan Repayment 
Program 

NHSC/CA Community Scholarship 
Program for Physician Assistants and 
Nurse Practitioners 


V 

V 


Delia Santiago 

Office of Statewide Health Planning & 
Development 
1 600 9th Street, Room 440 
Sacramento, CA 95814 
(916) 654-1833 


CO 


CO Health Professions Loan 
Repayment Program 


V 


T.R. Kautsky 
Colorado AHEC System 
UCHSC, Box A-096 
4200 E. 9th Avenue 
Denver, CO 80262 
(303)315-5885 
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STATE 


PROGRAMS 


NHSC/STATE 

FUNDED: 

EVALUATION 

REQUIRED 


CONTACTS 


CT 


CT Loan Repayment Program 


V 


Jann Moody 
Dept, of Public Health 
999 Asylum Avenue 
Hartford, CT 061 05 
(860) 509-8051 


DE 


Scholarship Program (service 
commitment) 

DIMER 




Marilyn Quinn 

Higher Education Commission 
Carvel State Office Building 
820 N. French Street 
Wilmington, DE 19801 
(302) 577-3240 


FL 


FL Health Service Corps Loan Repayment 
Program 

Nursing Loan Forgiveness Program 
Nursing Scholarship Program 


V 


Lat Penland 

HRS Health Professional Recruitment 
1317 Winewood Blvd. 
Tallahassee, FL 32399-0700 
(800) 342-8660 

Michelle Williamson 
Univ. of S. Florida College of Medicine 
12901 Bruce B. Downs Blvd., MDC 
Box 4 

Tampa, FL 33612-4799 
(813)974-2068 


CA 


GA Scholarship Program 


V 


Joe Lawley 

State Medical Education Board of 




GA Loan Repayment Program 


V 


Georgia 

244 Washington Street, S.W., Room 
574J 

Atlanta, GA 30334 
(404) 656-2226 


ID 


ID Health Professional Loan Repayment 
Program 

ID Community Scholarship Program 


V 


Debbie Braun and Barbara 
Cunningham 

Institute of Rural Health Studies 
Campus Box 8174 
Pocatello, ID 83209 
(208) 236-4436 


IL 


Medical Student Scholarship program 

Allied Health Care Professional 
Assistance Program 

IL National Health Service Corp Loan 
Repayment Program 


V 


Tom Yocom 
Center for Rural Health 
535 West Jefferson Street 
Springfield, IL 62761 
(217) 782-1624 
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STATE 


PROGRAMS 


NHSC/STATE 

FUNDED: 

EVALUATION 

REQUIRED 


CONTACTS 


IN 


Primary Care Scholarship Program 
Nursing Scholarship Fund Program 




Fred L. Ficklin 

Indiana Univ. School of Medicine 
635 Barnhill Drive, MS Room 162 
Indianapolis, IN 46202 
(317) 274-1965 
Yvonne Heflin 

State Student Assistance Commission 
1 50 W. Market, 5th Floor 
Indianapolis, IN 46204 
(317)232-2350 


IA 


IA Loan Repayment Program 


V 


Carl Kulczyk 
IA Dept, of Public Health 




IA Community Scholarship Program 


V 


321 E. 12th Street 
Des Moines, IA 50319 
(515) 281-7223 


KS 


KS Medical Student Loan Program 




Billie Jo Hamilton 

University of Kansas Medical Center 
3901 Rainbow Blvd. 

Kansas City, KS 661 60 
(913) 588-5170 


KY 


Rural KY Medical Scholarship Fund 
Establish Practice Grant Program 

KY Community Scholarship Program 


V 


Becky Vincent 

Rural Kentucky Medical Scholarship 
Fund 

301 N. Hurstbourne Parkway, Suite 200 
Louisville, KY 40222 
(502) 426-6200 

JoAnn Meyers 
Center for Rural Health 
University of Kentucky 
1 00 Airport Gardens Road, Suite 1 0 
Hazard, KY 41701 
Phone: 606.439.3557 


LA 


State Loan Repayment Program for 
Physicians, Dentists & Midlevels 


V 


Beth Millet 

LA Dept, of Health and Hospitals 
P.O. Box 1349 

Baton Rouge, LA 70821-5169 
(504) 842-4702 


ME 


Loan Repayment Program 


V 


Sophie Glidden 
Dept, of Human Services 




ME Community Scholarship Program 
Postgraduate Health Professions Program 


V 


1 1 State House Station 
Augusta, ME 04333 
(207) 624-5424 

Gloria Nadeau 
Finance Authority of Maine 
1 Weston Court 
1 19 State House Station 
Augusta, ME 04333 
(207) 626-8200 
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STATE 


PROGRAMS 


NHSC/STATE 

FUNDED: 

EVALUATION 

REQUIRED 


CONTACTS 


MD 


Loan Repayment Program 

Family Practice Scholarship Program 

Physician & Oc Ther and Asst. Grant 
Program 

State Nursing Scholarship & Living 
Expenses Grant 

Loan Asst. Repayment Program for 
Primary Care Physicians 


V 


Linda Weippert 

MD Higher Education Commission 
1 6 Francis Street 
Annapolis, MD 21 401 -1 781 
(410) 974-5370 x. 157 


MA 


State Loan Repayment Program 


V 


Rachel Gunther 
MA Dept, of Public Health 
250 Washington Street, 5th Floor 
Boston, MA 02108 
(617) 624-6066 


Ml 


Ml Loan Repayment Program 


V 


Steven Creamer 

Michigan Department of Community 
Health 

3423 N. MLK Blvd., P.O. Box 30195 
Lansing, Ml 48909 
(517) 335-8553 


MN 


Nurse Loan Repayment Program 

Rural Mid Level Practitioner Loan 
Repayment Program 

Rural Physician Loan Repayment 
Program 

Urban Physician Loan Repayment 
Program 

MN State Loan Repayment Program 


V 


Debra L. Jahnke 
MN Dept, of Health 
Office of Rural Health and Primary 
Care 

P.O. Box 64975 
St. Paul, MN 55164 
(612) 282-6334 


MS 


State Medical Education 
Loan/Scholarship Program 




Dottie C. Strain 
Institutions of Higher Learning 
3825 Ridgewood Road 
Jackson, MS 39211-6453 
(601)982-6663 


MO 


Scholarship Loan Program 

Primary Care Resource Initiative for MO 

MO Professional and Practical Nursing 
Student Loan and Loan Repayment 
Programs 


V 


Harold Kirbey 
MO Dept, of Health 
P.O. Box 570 
Jefferson City, MO 65102 
(800) 891-7415 
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STATE 


PROGRAMS 


NHSC/STATE 

FUNDED: 

EVALUATION 

REQUIRED 


CONTACTS 


MT 


MT Rural Physician Incentive Program 




Rod Sundsted 
MT University System 
Office of the Commissioner of Higher 
Education 
2500 Broadway 
Helena, MT 59620-3101 
(406) 444-6570 


NE 


NE Loan Repayment Program 

NE Medical and Physician Assistant 
Student Loan Program 




Kay Pinkley 

Nebraska Office of Rural Health 
301 Centennial Mall South, P.O. Box 
95007 

Lincoln, NE 68509-5007 
(402)471-2337 


NV 


NV Community Scholarship Program 
NV Loan Repayment Program 


V 


Caroline Ford 

Univ. of Nevada School of Medicine 
Center for Education and Health 
Services Outreach, MS 150 
Reno, NV 89557 
(702) 784-4841 


NH 


Federal/State Loan Repayment 
State Loan Repayment 


V 


John D. Bonds 

NH Dept, of Health and Human 
Services 
6 Hazen Drive 
Concord, NH 03301 
(603) 271-4617 


NJ 


Primary Care Physician & Dentist Loan 
Redemption Program 


V 


Sharon Bryant 

Primary Care Physician and Dentist 
Loan Redemption Program of New 
Jersey 

65 Bergen Street, 1 4th Floor 
Newark, NJ 07107-3000 
(201 ) 982-4605 


NM 


Osteopathic Medical Student Loan for 
Service Program 

Medical Student Loan-for-Service 
Program 

Allied Health Student Loan-for-Service 
Program 

Nursing Student Loan-for-Service 
Program 

Health Professional Loan Repayment 
Program 




Kenn Williams 

Financial Aid and Student Services 
P.O. Box 15910 
Sante Fe, NM 87506-5910 
(505) 827-7383 
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STATE 


PROGRAMS 


NHSC/STATE 

FUNDED: 

EVALUATION 

REQUIRED 


CONTACTS 


NY 


Regents Physician Loan Forgiveness 
Award Program 

Primary Care Service Corps 

Physician Loan Repayment Program 

Resident Loan Repayment Program 


V 


Douglas P. Mercado 
NY State Education Dept. 
Albany, NY 12234 
(518)486-5202 

Tom Kaczmarek 
NY State Dept, of Health 
Bureau of Health Resources 
Development 




Community Scholarship Program 


1 602 Corning Tower Building 
Albany, NY 12237-0053 




State Loan Repayment Program 


V 


(518)473-7019 


NC 


NC Student Loan Program for Health, 
Science & Mathematics 




NC Student Loan Program for Health, 
Science & Mathematics 
P.O. Box 20549 




Loan Repayment Program 
High Needs Service Bonus Program 
Residency Loan Program 


V 


Raleigh, NC 27619-0549 
(919) 571-4178 

Judi Ashbaugh 

NC Office of Rural Health and 
Resources Dev. 

31 1 Ashe Avenue 
Raleigh, NC 27606 
(919) 733-2040 


ND 


Physician Loan Repayment Program 

Midlevel Practitioner Loan Repayment 
Program 

ND Community Scholarship Program 


V 


Mary Amundson 
UND Center for Rural Health 
P.O. Box 9037 

Grand Forks, ND 58202-9037 
(701) 777-3848 


OH 


Physician Loan Repayment Program 




Susan Ewing-Ramsay 
OH Dept, of Health 
246 N. High Street, P.O. Box 118 
Columbus, OH 43266-0118 
(614) 466-3543 


OK 


Rural Medical Education Scholarship 
Loan Program 

Resident Rural Scholarship Loan Program 

Physician Community Match 

Nursing Student Assistance 
Scholarship Program 




James Bishop 

Physician Manpower Training 
Commission 

1 140 Northwest 63rd Street, Suite 302 
Oklahoma City, OK 73116 
(405) 843-5667 
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STATE 


PROGRAMS 


NHSC/STATE 

FUNDED: 

EVALUATION 

REQUIRED 


CONTACTS 


OR 


OR Rural Health Services Program 
OR Nursing Loan 

OR Community Scholarship Program 


V 


James Beyer 

OR State Scholarship Commission 
1500 Valley River Drive, Suite 100 
Eugene, OR 97401 
(541 ) 687-7385 


PA 


National Health Service Corps State Loan 
Repayment Program 

PA State Loan Repayment Program 


V 


Joseph B. May, III 
PA Dept, of Health 
Room 709, H & W Building 
Harrisburg, PA 17120 
(717) 772-5298 


Rl 


Rl Loan Repayment Program 


V 


Dolores Diorio 
Rl Dept, of Health 
3 Capitol Hill 
Providence, Rl 02908 
(401)277-3442 


sc 


SC State Incentive Grant for Advanced 
Level Practitioners 

SC Resident Incentive Grant 

SC State Incentive Grant Physicians 




Rebecca S. Seignious 
SC AHEC Recruitment & Retention 
Program 

1 71 Ashley Avenue 
Charleston, SC 29425 
(803) 771-2810 


SD 


SD Physician Tuition Reimbursement 
Program 

SD Midlevel Reimbursement Program 




Bart Hallberg 
SD Office of Rural Health 
445 E. Capitol 
Pierre, SD 57501 
(605) 773-4440 


TN 


Health Access Incentive Program 




Bill Jolley 

Office of Rural Health 
5th Floor, Cordell Hull Building 
426 Fifth Avenue North 
Nashville, TN 37247 
(615) 741-0388 


TX 


Physician Education Loan Repayment 
Program of TX 

State Medical Education Board 
TX Community Scholarship Program 


V 

V 


Bob Kirk 

Texas Higher Education Coordinating 
Board 

P.O. Box 12788 
Austin, TX 78711 
(512) 427-6367 




Rural Physician Assistant Reimbursement 
Program 

Outstanding Rural Scholar Recognition 
Program 




Carol Peters 

Center for Rural Health Initiatives 
21 1 E. 7th Street, Suite 91 5 
Austin, TX 78701 
(512) 479-8891 
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STATE 


PROGRAMS 


NHSC/STATE 

FUNDED: 

EVALUATION 

REQUIRED 


CONTACTS 


UT 


Rural Medical Education Scholarship 
Program 

Rural Medical Education Loan Re- 
payment Program 


V 


Robert J. Quinn 
UT Dept, of Health 
Box 142856 

Salt Lake City, UT 84114-2856 
(801) 538-6113 


VT 


VT State Loan Repayment Program 


V 


Ellen Thompson 
VT Dept, of Health 
1 08 Cherry Street 
P.O. Box 70 
Burlington, VT 05402 
(802) 863-7606 


VA 


Loan Repayment Program 
VA Medical Scholarship Program 


V 


Lilia M. Williams 
VA Dept, of Health 
Office of Primary Care Development 
1500 E. Main Street, Suite 213 
Richmond, VA 23219 
(804) 786-4891 


WA 


WA State Health Professional Loan Re- 
payment and Scholarship Program 


V 


Kathy McVay 

Higher Education Coordinating Board 
91 7 Lakeridge Way, S.W. 

P.O. Box 43430 
Olympia, WA 98504-3430 
(360) 753-7850 


wv 


WV Community Scholarship Program 

Health Sciences Scholarship Program 

Medical Student Loan Program 

State Loan Repayment Program 

Recruitment and Retention Community 
Project 


V 

V 


Jodie Jackson 
Office of Rural Health 
West Virginia University 
P.O. Box 9003 
Health Sciences North 
Morgantown, WV 26506-9003 
(304) 293-6753 

Alicia Tyler 

University System of West Virginia 
1018 Kanawha Blvd. East, Suite 1100 
Charleston, WV 25301-2827 
(304) 558-0530 


Wl 


Wl Physician Loan Assistance Program 

Wl Health Care Provider Loan Assistance 
Program 




Jane Thomas 

Wisconsin Department of Commerce 
123 West Washington Ave, P.O. Box 
7970 

Madison , Wl 53707 
(608) 267-3837 


Total 


1 1 1 Programs 


41 





NOTE: As a condition for receiving funds from the National Health Service Corp., these 41 Community Scholarship 
and State Loan Repayment Programs must turn in quarterly reports that contain evaluative program data. 
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Appendix l-C. Evaluations of State Scholarship and Loan Programs 



Comments 




Interview conducted. 


Interview conducted. 


Funded by federal and a nonprofit. 

The Community Scholarship Program will not 
reapply for funding from NHSC, but will monitor its 
recipients for the next two years. 


Funded by federal government and community. 




Interview conducted. 




Other 
















c 

o 

'rS 


No 

Money 










s 






> 

UJ 

O 

Z 


No Time 
or Staff 


















Too 

New 


s \ 














Planning 

Evaluation 


No 

No 


No 

No 


No 

No 


No 

No 


No 


No 


No 

No 


Evaluation 


No 

No 


Yes 

No 


Yes 

No 


No 

No 


No 


No 


No 

Yes 


Program 


Community Scholarship Program 
(4) 

Board of Medical Scholarship 
Awards 


AZ Medical Student Loan Pro- 
gram (1 ft) 

AZ Loan Repayment Program (2) 


AR Rural Medical Practice 
Student Loan/Scholarship Pro- 
gram (47) 

Community Match Student Loan 
& Scholarship Program (1) 


NHSC/CA State Loan Repay- 
ment Program (5) 

NHSC/CA Community Schol- 
arship Program for Physician 
Assistants and Nurse Practitio- 
ners 


CO Health Professions Loan 
Repayment Program (5) 


CT Loan Repayment Program (5) 


Scholarship Program (service 
commitment) (3) 

DIMER (27) 


State 


_i 

< 


i 

ZV 


AR 


CA 


CO 


CT 


DE 
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CO 
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Appendix l-D. Program Evaluation Tables 



Arizona 



Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 


AZ Medical Student Loan Program 


Year Program became Operational 


1978 


IMPETUS FOR EVALUATION 
First Evaluation 


1995 


Reason for Conducting Evaluation 


Legislative Mandate 


PROCESS 

Information Collected 


Cost of medical education; number of students 
enrolled in the program; state appropriations to the 
program; amount of loan to each student; number of 
physicians in service; physicians in service versus 
repayment; estimated number of physicians in the 
program available for service; placement, 
appropriations, students' stipends and number of 
students in the program compared with programs in 
six other states 


System of Data Collection 


Has tracked students using WordPerfect, but is now 
switching to Access software. 


Cost and Personnel Time 




240 hours 


Frequency of Updates 


Bi-annually 


Advice to Other Program Administrators 


Collect data that go beyond what the legislature 
requires (if appropriate); for example, the legislature 
has now mandated the program administrator to 
collect data on those providers who have served 
beyond their service obligation period and other 
such data. 


OUTCOMES 

Usefulness of Program Evaluation 


It was useful in that the program administrator 
gained an understanding of how the program has 
worked and realized how outdated the program had 
become. 


Multi-State Meeting on Lessons Learned from 
Program Evaluation 


Very useful, as long as funding for travel is provided. 
There is no nationwide program for state funded 




programs. 
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OTHER COMMENTS 


The 1 995 law identified 1 0 programs to be reviewed 
in the first cycle; this program was one of them. The 
program must present data bi-annually as part of the 
appropriations process. The program administrator, 
however, was unsure when she would have to 
present another in-depth Program Authorization 
Review to the legislature. 


CONTACT 


Maggie Gumble 

College of Medicine Financial Aid 
University of Arizona 
1 501 N. Campbell 
Tucson, AZ 85724 
(602) 626-7145 
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Arkansas 



Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 


The Arkansas Rural Medical Practice Student Loan 
and Scholarship Program 


Year Program became Operational 


1949 


IMPETUS FOR EVALUATION 
First Evaluation 


In 1995, the program evaluation was conducted 
because of a legislative mandate. Prior to that, it was 
evaluated as requested. 


Reason for Conducting Evaluation 


Legislative mandate 


PROCESS 

Information Collected 


Percent and number of recipients from 1 980 to 1 993 
that have complied with service, that did not comply 
but repaid the state with cash, that are pending and 
that are practicing in the state. 


System of Data Collection 


Compliance forms mailed to providers after their 
residencies; information collected manually. 


Cost and Personnel Time 


1 week 


Frequency of Updates 


Annually 


Advice to Other Program Administrators 


Evaluations are much easier to conduct if you keep 
good records throughout the years and know what 
report format to input the information. 


OUTCOMES 

Usefulness of Program Evaluation 


Too soon to tell. The evaluation was conducted last 
month, and they have not heard from the legislature. 


Multi-State Meeting on Lessons Learned from 
Program Evaluation 


Maybe, a meeting would be useful. The program 
evaluation requirements are already in law, but 
maybe they could propose changes to the law. 


OTHER COMMENTS 


None 


CONTACT 


Tom South 

College of Medicine Student & Academic Affairs 

University of Arkansas for Medical Sciences 

Financial Aid - Slot 709 

4301 W. Markham Street 

Little Rock, AR 72205-7199 

(501) 686-5813 
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Delaware 







Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 


Delaware Institute of Medical Education and 
Research (DIMER) 


Year Program became Operational 


1969 


IMPETUS FOR EVALUATION 
First Evaluation 


27th year of program 


Reason for Conducting Evaluation 


Recommended by the DE General Assembly Joint 
Sunset Committee 


PROCESS 

Information Collected 


Scope of state's primary care needs using a survey of 
primary care physicians conducted by the Division 
of Public Health; existing medical education 
opportunities for DE residents; and systematic review 
of DIMER operations. 


System of Data Collection 


Comprehensive review by the DE Health Care 
Commission Primary Care Committee 


Cost and Personnel Time 


No direct cost to DIMER because it was a legislative 
review. 


Frequency of Updates 


One time only 


Advice to Other Program Administrators 


Not applicable 


OUTCOMES 

Usefulness of Program Evaluation 


Evaluation led to the reauthorization of DIMER, 
expansion of its role and focus and an administrative 
move to the DE Health Care Commission. 


Multi-State Meeting on Lessons Learned from 
Program Evaluation 


Not applicable 
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OTHER COMMENTS 


Through DIMER, Jefferson Medical College of 
Thomas Jefferson University functions as the state's 
medical school since the state does not have one. 
DIMER grants and loans are offered to students as 
they enter Jefferson Medical College and carry a 
financial consequence of not returning to the state to 
practice. 


CONTACT 


Marilyn Quinn 

Higher Education Commission 
Carvel State Office Building 
820 N. French Street 
Wilmington, DE 19801 
(302) 577-3240 
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Idaho 



Categories 


Program Specific Information 


PROGRAM INFORMATION 




Program Name(s) 


Idaho Health Professional Loan Repayment Program 


Year Program became Operational 


1992 


IMPETUS FOR EVALUATION 
First Evaluation 


Fourth year of the program. 


Reason for Conducting Evaluation 


Requirement from Dept, of Health and Welfare. 


PROCESS 

Information Collected 


Where participants were raised and where they plan 
to be practicing (rural vs. urban areas), new 
programs participants have implemented in their 
communities, the communities' thoughts on the 
program's responsiveness to inquiries, and their 
willingness to write to legislators in support of more 
funding for the program. 


System of Data Collection 


Two different mail surveys— one to participants and 
the other to the Primary Care Service Areas-which 
were followed-up by phone and fax. 


Cost and Personnel Time 


No cost because volunteers conducted the surveys; 
two weeks. 


Frequency of Updates 

Advice to Other Program Administrators 


Depends on the availability of funds. 


Try to recruit students for conducting phone surveys. 


OUTCOMES 

Usefulness of Program Evaluation 


Evaluations are a good justification for a program's 
existence and for asking the legislature for more 
funding. 


Multi-State Meeting on Lessons Learned from 


This meeting would be valuable given that travel 


Program Evaluation 


scholarship funds are provided. 


OTHER COMMENTS 


Rather than reinvent the wheel, let's tap resources 
that are available by learning from other state 
scholarship and loan programs. 


CONTACT 


Debbie Braun 

Institute of Rural Health Studies 
Campus Box 81 74 
Pocatello, ID 83209 
(208) 236-4436 
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Illinois 



Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 


Medical Student Scholarship Program 


Year Program became Operational 


1978 


IMPETUS FOR EVALUATION 
First Evaluation 


1984-1985 academic year 


Reason for Conducting Evaluation 


Legislative mandate 


PROCESS 

Information Collected 


Students receiving new and continuing scholarships; 
expenditures and fiscal history of program; number 
of scholarships by public and private medical 
schools; ethnicity and gender of recipients; retention 
rates; recipients' medical specialty and geographic 
distribution; recipients' practice location; type of 
high school of those serving beyond obligation; 
academic failures; monetary repayment; and number 
of buyouts by school. 


System of Data Collection 


Use Data Ease software and mail surveys every six 
months on prepared forms to track students, 
residents and those in practice. 


Cost and Personnel Time 


Two weeks; $$ unknown 


Frequency of Updates 


Annually - every March 


Advice to Other Program Administrators 


Think about who wants to know what and what 
information is useful before you evaluate the 
program; use a user-friendly, manageable data base 
to track students, so anyone can use it; identify other 
programs run by universities, higher education 
boards and rural health departments that are also 
channeling providers into underserved areas; include 
a comments section in the evaluation; track 
recipients' years of service in the communities. 


OUTCOMES 

Usefulness of Program Evaluation 


It has given us confidence in that we can 
demonstrate the effectiveness of the program with 
hard numbers. 


Multi-State Meeting on Lessons Learned from 
Program Evaluation 


Valuable. Ideas for dicussion include reciprocity, 
litigation and contract designs. Keep the meeting 
small and regional. 
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OTHER COMMENTS 




CONTACT 


Tom Yocom 
Center for Rural Health 
535 West Jefferson Street 
Springfield, IL 62761 
(217) 782-1624 
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Michigan 



Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 


State Loan Repayment Program 


Year Program was Established 


1990 


IMPETUS FOR EVALUATION 
First Evaluation 


Program in existence for five years when first 
evaluated. 


Reason for Conducting Evaluation 


Want to know status of retention situation of loan 
recipients. Had the resource availability of student 
intern. 


PROCESS 

Information Collected 


Retention status of loan recipients. 


System of Data Collection 


Telephone survey; hand tabulation of responses. 


Cost and Personnel Time 


$300; one week. 


Frequency of Updates 


Expect to update in 1 997 (two years since the last 
survey); hope to do annually thereafter. 


Advice to Other Program Administrators 


Develop data base of all recipient practice sites. 
Collect data via mail questionnaire. 


OUTCOMES 

Usefulness of Program Evaluation 


Very useful, particularly as time goes on. Data used 
to support federal loan repayment grant, provide 
anecodotes to legislators on value of program, recruit 
new participants by providing data on prospective 
sites. 


Multi-State Meeting on Lessons Learned from 
Program Evaluation 


1 do not know. Limit of state travel funds also a 
problem. 


OTHER COMMENTS 


None. 


CONTACT 


Steven Creamer 

Michigan Department of Community Health 
3423 N. MLK Blvd., P.O. Box 30195 
Lansing, Ml 48909 
(517)335-8553 
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Nebraska 







Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 


Rural Health Scholarship Program 


Year Program became Operational 


1979 


IMPETUS FOR EVALUATION 
First Evaluation 

Reason for Conducting Evaluation 


Program in existence for 14 years when first 
evaluated. 




Offered the free resources of a graduate student. 


PROCESS 

Information Collected 


Demographic analysis of program participants who 
had completed their service obligation to categorize 
and analyze retention successes and practice plans. 
Comparisons of location of residency to retention in 
obligated service areas. 


System of Data Collection 


Manual tabulation of existing participant data base; 
no interviews performed. 


Cost and Personnel Time 


60 to 80 hours 


Frequency of Updates 


None; future systematic evaluations subject to the 
availability of funds. 


Advice to Other Program Administrators 


Conduct participant interviews to gauge feelings of 
the program's influence on training and practice 
locations. 


OUTCOMES 

Usefulness of Program Evaluation 


Useful in knowing relationship between where 
participant did residency training and where he/she 
completed service obligation. 


Multi-State Meeting on Lessons Learned from 
Program Evaluation 


Valuable to have such a meeting; would be wifling 
to participate. 


OTHER COMMENTS 


None. 


CONTACT 


Kay Pinkley 

Nebraska Office of Rural Health 
301 Centennial Mall South, P.O. Box 95007 
Lincoln, NE 68509-5007 
(402)471-2337 
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New Jersey 



Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 


Primary Care Physician and Dentist Loan 
Redemption Program 


Year Program became Operational 


1992 


IMPETUS FOR EVALUATION 
First Evaluation 


Program in existence for two years when first 
evaluated. 


Reason for Conducting Evaluation 


Health department evaluated many programs, 
including theirs. 


PROCESS 

Information Collected 


Recruitment; best practices from programs in CT, FL, 
LA, MA, ME, MD, NY, TX and WA; practice sites; 
patient and practitioners' satisfaction with the 
program; provider sensitivity to community needs. 


System of Data Collection 


Used outside research firm, MRH Evaluations, to 
conduct study. 


Cost and Personnel Time 


Roughly $10,000; eight months. 


Frequency of Updates 


Subject to the availability of funds. 


Advice to Other Program Administrators 


Evaluation was very worthwhile. 


OUTCOMES 

Usefulness of Program Evaluation 


Very useful. Recommendations from evaluation 
were incorporated into a proposal to the governor. 


Multi-State Meeting on Lessons Learned from 
Program Evaluation 


Valuable to have such a meeting; would be willing 
to participate given travel funds are provided. 


OTHER COMMENTS 


States should conduct site visits, as well as phone 
surveys, to collect program evaluation information. 


CONTACT 


Sharon Bryant 

Primary Care Physician and Dentist Loan 
Redemption Program of New Jersey 
65 Bergen Street, 14th Floor 
Newark, NJ 07107-3000 
(201) 982-4605 
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Oklahoma 







Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 

Year Programs were Established 


1) Rural Medical Education Scholarship Program 

2) Family/General Practice Resident Rural 
Scholarship 

3) Physician Community Match Loan Program 

4) Nursing Student Assistance Scholarship Program 




1) 1975; 2) 1992; 3) 1992; 4) 1982 


IMPETUS FOR EVALUATION 
First Evaluation 

Reason for Conducting Evaluation 


Programs in existence for one year when first 
evaluated. 




Legislative mandate and report to the governor. 


PROCESS 

Information Collected 


Provider practice location and specialty, length of 
time in obligation, retention status. 


System of Data Collection 


Annual mail questionnaire form tabulated on 
computer data base. 


Cost and Personnel Time 


One day of work. 


Frequency of Updates 


Annually. 


Advice to Other Program Administrators 
OUTCOMES 


Keep the evaluation process simple. Don't rely on 
licensure or medical society records to locate 
providers. 


Usefulness of Program Evaluation 


Able to annually decide whether strategies need to 
be fine tuned. 


Multi-State Meeting on Lessons Learned from 
Program Evaluation 


Valuable to have such a meeting; would be willing 
to participate. 


OTHER COMMENTS 


None. 


CONTACT 


James Bishop 

Physician Manpower Training Commission 
1 140 Northwest 63rd Street, Suite 302 
Oklahoma City, OK 73116 
(405) 843-5667 
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Tennessee 



Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 


Health Access Incentive Program 


Year Program was Established 


1989 


IMPETUS FOR EVALUATION 
First Evaluation 

Reason for Conducting Evaluation 


Program in existence for seven years when first 
evaluated. 




Request from new Commissioner of Health. 


PROCESS 

Information Collected 


Placement by specialty and location. Retention status 
of grant recipients. 


System of Data Collection 
Cost and Personnel Time 


Computer data base tabulating results of annual 
survey of all primary care physicians in rural areas of 
state. 


Frequency of Updates 


One FTE analyst, three part-time regional staff. Will 
contract with university for $30,000 to perform 
evaluation of physician assistants and advanced 
practice nurses. 


Advice to Other Program Administrators 


Annually 




Keep questions short and concise; use tool to ensure 
high response rate. Obtain data on FTE levels so if 
actual levels are lower than previously known in a 
given area, then such areas may qualify for federal 
HPSA or MUA status. 


OUTCOMES 

Usefulness of Program Evaluation 

Multi-State Meeting on Lessons Learned from 


Results helpful in deciding to restructure the program 
next year to focus on more of a local health care 
system's needs rather than look at individual 
provider placements. (Would many providers 
already have gone to such communities anyway?) 
Regions will be given more flexibility to design 
business plans. 


Program Evaluation 


Valuable to have such a meeting; would be willing 
to participate. 


OTHER COMMENTS 


None. 
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CONTACT 



Bill Jolley 

Office of Rural Health 
5th Floor, Cordell Hull Building 
426 Fifth Avenue North 
Nashville, TN 37247 
(61 5) 741-0388 
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Texas 







Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 


Physician Education Loan Repayment Program 


Year Program was Established 


1987 


IMPETUS FOR EVALUATION 
First Evaluation 


Program in existence for five years when first 
evaluated. 


Reason for Conducting Evaluation 


Provide additional justification to receive federal 
loan repayment matching funds. To obtain useful 
data. 


PROCESS 

Information Collected 


Loan recipient perceptions of the importance of 
PELRP in deciding to practice in HPSA and 
remaining there; retention data. 


System of Data Collection 


Internally administered survey gauging simplified 
responses that are easily tabulated. 


Cost and Personnel Time 


Roughly $1,300; two weeks. 


Frequency of Updates 


First survey performed in 1992; hope to conduct 
similar evaluation in a few years. 


Advice to Other Program Administrators 


Keep surveys simple and easily comparable to 
previous and future evaluations. Beware of 

respondent bias - those with the most vested interest 
will respond first. 


OUTCOMES 

Usefulness of Program Evaluation 

Multi-State Meeting on Lessons Learned from 


Very useful. Able to establish trendline and compare 
with previous surveys. Can provide insight to what 
extent recipients already had interest and intention in 
serving in HPSA; would they have gone there 
anyway? 


Program Evaluation 


Valuable to have such a meeting; would be willing 
to participate given travel funds are provided. 


OTHER COMMENTS 


None. 
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Texas 



Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 


Outstanding Rural Scholar Recognition Program; 
Community Scholarship Program 


Year Program became Operational 


1992, 1994 


IMPETUS FOR EVALUATION 
First Evaluation 


First year of programs. Since 1 987, the Center for 
Rural Health Initiatives has published a report on 
Rural Health in Texas that contain evaluative data on 




the scholarship programs among other types of data 
such as activities of the center and the availability of 
health professionals. 


Reason for Conducting Evaluation 


Legislative Mandate 


PROCESS 

Information Collected 


Number of communities that have sponsored 
students; number of students in the programs; 
recruitment and retention data. 


System of Data Collection 


Annual letter to sponsors 


Cost and Personnel Time 


Four months to create the entire rural health report 
(not just the cumulative data on scholarship 
recipients). 


Frequency of Updates 


Bi-annually 


Advice to Other Program Administrators 


Form close liaisons with staff in related agencies to 
receive relevant data that is critical to the program 
such as HPSA data. 


OUTCOMES 

Usefulness of Program Evaluation 


Yes. A few legislators from rural areas use the report 
to maintain funding for rural projects and 
appropriate small, additional amounts to existing 




programs. 


Multi-State Meeting on Lessons Learned from 
Program Evaluation 


It would be a good idea, particularly for those that 
are trying to develop or expand their programs. It's 
always valuable to learn from other states. 


OTHER COMMENTS 




CONTACT 


Carol Peters 

Center for Rural Health Initiatives 
211 E. 7th Street, Suite 91 5 
Austin, TX 78701 
(512)479-8891 
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West Virginia 



Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 


Health Sciences Scholarship Program 


Year Program became Operational 


1995 


IMPETUS FOR EVALUATION 
First Evaluation 


First year of program 


Reason for Conducting Evaluation 


Request from the vice chancellor for health sciences 


PROCESS 

Information Collected 
System of Data Collection 


Students who were born in state; students who have 
completed WV Rural Education Partnerships 
rotations or have other rural training; areas of 
intended practice for medical students participating 
in the program. 


Cost and Personnel Time 


Gathers this information from students' applications 
forms. An independent firm is developing a custom- 
made data base for the program. 


Frequency of Updates 


N/A 


Advice to Other Program Administrators 


Ongoing once the data base is created. 


OUTCOMES 

Usefulness of Program Evaluation 


Yes. It is important to go beyond financial 
information and collect outcome data. 


Multi-State Meeting on Lessons Learned from 
Program Evaluation 


Great idea. We need more face-to-face contact 
especially as we are all trying to do more with less. 


OTHER COMMENTS 


Higher education and health sciences schools must 
collect data for a report card that is issued to the 
legislature. However, the report card focuses on 
financial information, instead of outcomes. 


CONTACT 


Alicia Tyler 

University System of West Virginia 
1018 Kanawha Blvd. East, Suite 1100 
Charleston, WV 25301-2827 
(304) 558-0530 
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Wisconsin 



Categories 


Program Specific Information 


PROGRAM INFORMATION 
Program Name(s) 


Physician Loan Assistance Program 


Year Program became Operational 


1991 


IMPETUS FOR EVALUATION 
First Evaluation 

Reason for Conducting Evaluation 


Program in existence for one year when first 
evaluated. 

Was new program doing what it was designed to do? 
No mandate to do it. 


PROCESS 

Information Collected 


Recipient demographics including location/nature of 
training, specialty, previous exposure to underserved 
area and other factors influencing practice in such 
settings, time in obligation, amount of loan, current 
patient workload. Value of PLAP in recruitment. 
Program process issues. Other factors to help 
improve provider recruitment in underserved areas. 


System of Data Collection 


Hand tabulated data base on recipients. 


Cost and Personnel Time 


35 FTE (two people) over four months. 


Frequency of Updates 


Not updated since 1992. 


Advice to Other Program Administrators 


Conduct formal research project; make sure data 
base is sound by ensuring loan application contains 
most information of what will be needed. Survey 
should be performed periodically. 


OUTCOMES 

Usefulness of Program Evaluation 


Very useful, but needs updating. Inform legislators 
of the value of state dollars. Program has made a 
difference but is realization that program's impact 
may be marginal; many recipients would have gone 
to underserved areas anyway. 


Multi-State Meeting on Lessons Learned from 
Program Evaluation 


Valuable to have such a meeting; would be willing 
to participate. 


OTHER COMMENTS 


None 
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CONTACT 



Mark Shapleigh 

Wisconsin Office of Rural Health 
740 WARF, 61 0 Walnut Street 
Madison, Wl 53705-2336 
(608) 265-3603 
Jane Thomas 

Wisconsin Department of Commerce 
1 23 West Washington Ave, P.O. Box 7970 
Madison , Wl 53707 
(608) 267-3837 
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Appendix l-E. NHSC Reporting Requirements 



The reporting requirements are contained in four separate sections of the current statute and are reported in each 

annual report. 

Section 336A requires the following data: 

1. The number, identify and priority of all HPSAs designated during the year and the number of HPSAs which the 
secretary estimates will be designated in the subsequent year; 

2. The number of applications during the year for assignment of NHSC members and the action taken on each 
application; 

3. The number and types of NHSC members which the secretary estimates will be assigned to such areas in the 
subsequent year, and the need for additional members for the NHSC; 

4. The recruitment efforts engaged in for the NHSC during the year and the number of qualified individuals who 
applied for service in the NHSC during the year; 

5. The number of patients seen and the number of patient visits recorded during the year with respect to each HPSA 
to which an NHSC member was assigned during the year; 

6. The number of NHCS members who elected, and the number of NHCS members who did not elect, to continue 
to provide health services in HPSAs after termination of their service in the NHCS and the reasons for not making 
such election; 

7. The results of evaluations and determinations made under Section 333 during the year; and 

8. The amount charged during the year for health services provided by NHSC members, the amount which was 
collected during the year by appropriate entities and the amount paid to the secretary during the year under such 
agreements. 

Section 338A(i) requires an annual report for the previous fiscal year that includes: 

1. The number and type of health professions training of students receiving scholarships under the Scholarship 
Program; 

2. The educational institutions at which such students are receiving their training; 

3. The number of applications filed in the school year beginning in the reporting year and in prior school years; 

4. The amount of scholarship payments made for tuition, stipends, and other expenses, in the aggregate and at each 
educational institution for the reporting year and for prior school years; 

5. The number and type of health professions training of individuals who have breached the scholarship contract; 

6. The educational institutions attended by individuals who have breached the scholarship contract; 

7. The amounts for which individuals who have breached the scholarship contract are liable to the U.S.; 

8. The extent to which individuals who have breached the scholarship contract have paid the amounts for which 
they are liable to the U.S.; 

9. The basis for the decision of individuals who have breached the scholarship contract; and 
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10. The effectiveness of the Secretary in recruiting health professionals to participate in the scholarship program; and 
in encouraging and assisting such professionals to provide primary health service to HPSAs after they have 
completed their period of obligated service under the program. 

Section 338B(i) requires an annual report for the previous fiscal year that includes: 

1 . The total amount of the loan payments made under the Loan Repayment Program; 

2. The number of applications filed; 

3. The number and type of health professions training of individuals receiving loan repayments; 

4. The educational institutions at which such students are receiving their training; 

5. The total amount of the indebtedness of such individuals for educational loans as of the date on which the 
individuals became participants in such program; 

6. The number of years of obligated service specified for such individuals in the initial contracts, and, in the case of 
individuals whose period of such service has been completed, the total number of years for which the individuals 
served in the Corps; 

7. The number and type of health professions training of individuals who have breached the contract with respect 
to: the educational institutions with respect to which payments have been made or were to be made under 
contract; the amounts for which the individuals were liable to the U.S. under section 338E; the extent of payment 
by the individuals of such amounts; and if known, the basis for the decision of the individuals to breach the 
contract; and 

8. The effectiveness of the secretary in recruiting health professionals to participate in the Loan Repayment 
Program; and in encouraging and assisting such professionals to provide primary health service to HPSAs after 
they have completed their period of obligated service under the program. 

Section 338H(a) requires an annual report to the Committees on Labor and Human Resources (Senate), Energy and 

Commerce (House) and Appropriations (House and Senate) on:: 

1 . The number of health care providers who will be needed for the NHSC during the five fiscal years beginning 
after the date the report is filed; and 

2. The number of: scholarships the secretary proposed to provide under the Scholarship Program during such five 
fiscal year; the individuals for whom the secretary proposes to make loan repayments under the Loan Repayment 
Program during such five fiscal years; and the individuals who have no obligation and who the secretary 
proposes to have as members of the NHSC during such five fiscal years. 

Although not a report requirement, the NHSC has added the following information because the State Loan 

Repayment Program also provides services to residents of HPSAs: 

The number of providers the secretary proposes to fund through grants to states under the State Loan Repayment 

Program during such five fiscal years. 

Source: Department of Health and Human Services, Public Health Service, Health Resources and Services 

Administration, Bureau of Primary Health Care, National Health Service Corps, Report to the Congress for Years 

1990-1994. 
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Appendix l-F. Program Descriptions: Illinois and New Jersey 



The Medical Student Scholarship Program (MSSP), enacted into law in 1977, awards scholarships to state residents 
attending medical schools in Illinois. The program has undergone many changes throughout the years. Most 
significantly, the Center for Rural Health which administers the program has tightened the penalties for participant 
noncompliance. If a participant fails to practice in an underserved area, he/she now must pay three times the 
amount or 300 percent of the scholarship received. The State also has the ability to sanction the renewal of a 
recipient's license if he/she fails to meet their service obligation and pay penalties. 

To measure the effects of these changes and the program's overall effectiveness, the program has been required, by 
state law, to be evaluated annually since the 1984-1985 academic year. Each year, the program administrator 
updates information on the number of students receiving new and continuing scholarships; program expenditures 
and the fiscal history of the program; the number of scholarships by public and private medical schools; the ethnicity 
and gender of recipients; retention rates; the recipients' medical specialty and geographic distribution; the recipients' 
practice location; the type of high school of those serving beyond obligation; the number of academic failures; 
monetary repayment; and the number of buyouts by school. 

This information is collected by mailing surveys every six months on prepared forms to program participants that 
include students, residents and those in practice. The program administrator uses Data Ease, a software package, to 
enter the data, but he does not recommend it to other administrators. The evaluation takes only about two weeks of 
his time because he simply has to update the information in the report each year. 



By formally evaluating the program each year, the administrator has shown that these program modifications have 
helped make the program more successful. Funds for the program were renewed in 1985 after being suspended for 
four years. The evaluation is now showing signs that the program is working. In the first year of the program, 75 
percent of the recipients did not complete their medical education. Now 64 percent of the recipients stay at least 
one year beyond their obligation period. 

In 1993, the New Jersey Department of Health awarded MRH Evaluations, Inc. the contract to evaluate the 
effectiveness of the New Jersey Primary Care Physician and Dentist Loan Redemption Program (LRP) which was 
created in 1991 . The LRP evaluation was part of a comprehensive review of several new health initiatives that were 
authorized by the state legislature. 



After being in operation for 18 months, MRH Evaluations measured LRP's effectiveness in recruiting qualified 
providers, whether LRP reflected "best practice" for recruiting physicians and dentists to work in underserved areas; 
whether LRP providers were placed in areas of identified need; changes in patients' access to primary care as a result 
of service dispensed by the LRP providers; patients' and providers' satisfaction with the LRP; and providers' sensitivity 
to their community's cultural, ethnic, racial and socioeconomic characteristics. 



MRH Evaluations used several quantitative and qualitative strategies to collect data to document the LRP's operating 
features, the perceived satisfaction of those involved in the program and the program's impact. Specifically, 
researchers documented the following programmatic information: how providers learned about the LRP, how 

practice sites learned about the LRP, LRP recruitment practice, LRP default provisions, LRP loan amounts, LRP 
practice sites, patient populations served by practice sites, sources of income for practice sites, characteristics of non- 
LRP Medicaid providers, non-LRP providers by Medicaid claims, providers' ratings of LRP features, providers' level of 
satisfaction regarding placement, providers' experiences prior to LRP participation and providers' professional 
objectives. 



Base line information was obtained about the providers and practice-sites through surveys and existing program 
records. In-depth information was obtained through interviews of six practice sites, three providers and the staffs of 
the New Jersey Departments of Health and Higher Education, as well as LRP staff. 



The evaluation concluded that the LRP has accomplished many of its objectives. Overall, practice sites, providers 
and their patients were satisfied with the LRP. In addition, the report lists a number of recommendations. Many of 
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these recommendations have been incorporated into a proposal to the Governor. The program administrator hopes 
that these recommendations will be implemented in the near future. 
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Appendix ll-A. State Health Professions Education Legislation, 1985-1993 
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Source: Intergovernmental Health Policy Project, National Conference of State Legislatures, 
November 1996 
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Appendix ll-B. Evaluations of Recent State Health Professions Education Reforms 
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